Security holes in Intel CPUs

SHORT: This is my collection of links to articles about Intel CPU/chipset security holes.

For some longer time I’m interested in security holes/flaws in PC hardware as I find such things a real nightmare from software developer point of view. What good is a perfectly written software with a state of the art security if the hardware allows to bypass it?

Last years I’ve noticed that such a security flaw is continuously present in Intel CPUs in form of Intel ME and Intel AMT technology. Please let me know if similar findings exist for AMD CPUs.

Someone may say that security problems described in articles listed below are rather related to Intel chipsets than to Intel CPUs. However nowadays you can’t (even on a desktop computer) have an Intel CPU and a non-Intel chipset on your motherboeard (in old days it was possible: SiS chipsets, NVidia chipsets, etc.). So when choosing an Intel CPU you really choose an entire Intel platform (CPU, chipset, etc.) with all these problems. Thus this all begins with an Intel CPU – so is the title of this post.

Intel Management Engine (ME) / Intel Active Management Technology (AMT)

It looks like Intel ME/AMT is a hardware backdoor present in all Intel systems (CPU+chipset) since 2008 (introduction of Nehalem cores) or even earlier on systems with vPro technology. It’s a separate computer, able to execute arbirary code, able to control all buses in the “main” computer (the one user interacts with) and it’s working when there is a power supply connected (even when the “main” computer is turned off).

  1. Intel Management Engine (ME) – Libreboot FAQ
  2. A Quest To The Core. Thoughts on present and future attacks on system core technologies by Joanna Rutkowska – an overwhelming presentation of hardware holes (mainly in Intel chipsets and CPUs) and how thay can be exploited. (2009)
  3. Why Rosyna Can’t Take A Movie Screenshot – a nice article describing what this technology (Intel ME/AMT) can do. There is a lot of related links under the article. (2015)
  4. Intel x86 considered harmful – a paper by Joanna Rutkowska being a survey of the various problems and attacks presented against the x86 platform over the last 10 years. (2015)
  5. Intel x86s hide another CPU that can take over your machine (you can’t audit it), (2016)
  6. Intel AMT Vulnerability Shows Intel’s Management Engine Can Be Dangerous – Intel published a security advisory about a vulnerability in Intel ME/AMT. (2017)
  7. CVE-2017-5689“An authentication bypass vulnerability affecting just about every Intel server with AMT, ISM or Intel Small Business technology enabled, allowing unprivileged network attackers to gain system privileges (where AMT has been provisioned). This is notable because AMT provides the possibility to remotely control a computer even if when powered off. Packets sent to ports 16992 or 16993 are redirected through Intel’s Management Engine (a small, separate processor independent of the main CPU) and passed to AMT. Patch rollouts are expected to be slow, as while it is Intel’s responsibility to develop the patches (which it has done), it is not Intel’s responsibility to deliver them. That’s down to the device manufacturers and OEMs; and it is generally thought that not all will do so.”

Intel Processor Trace (PT)

  1. CyberArk: Windows 10 Vulnerable To Rootkits Via Intel’s Processor Trace Functionality, (2017)

Intel System Management Mode (SMM)

SMM was originally introduced by Intel so we can call it Intel technology. However it’s present in AMD CPUs as well.

  1. Most Intel x86 Chips Have a Security Flaw, (2015)
  2. SMM problems – summary on Wikipedia

Intel SYSRET

This time Intel’s implementation of a particular x86 instruction was worse that the one found in AMD CPUs.

  1. The Intel SYSRET privilege escalation, (2012)
Posted in BIOS, hardware, security | 2 Comments

Google App Engine, Java, JPA 2, Spring Framework, Maven

DISCLAIMER: This was going (in 2014!) to be a full tutorial but at some point I’ve lost motivation in digging in the Google App Engine. Actually I was so tired with this GAE data store that I felt really relieved when returned to using a classic relational database. The draft of this article was hanging for couple of years waiting to be completed. Now I’ve finally realized it will never be. As far as I recall the main problem with Google App Engine data store was related to using transactions. Probably things behave a little bit better if you… are not using transations. So be warned:

  • I publish this as a DRAFT just in case someone will find it usefull.
  • The state of knowledge about GAE here is somewhere in 2014

This tries to present how to create a Maven managed project of a Java web application to be deployed on Google App Engine and making use of Spring Framework and JPA 2 as a persistence layer. It shows how to store related entities (@OneToMany) in a transaction (key complication factor) and how to mark JPA entity as a child entity in sense of Google App Engine storage. The tutorial assumes some basic knowledge of Java, Maven, Spring Framework and JPA. I used following software versions:

  1. Google App Engine SDK, ver. 1.9.0
  2. Java SDK ver. 1.7, as required by the above version of Google App Engine
  3. JPA 2.0
  4. Spring Framework ver. 3.2.8
  5. Maven ver. 3.2.1

Create a project skeleton

Go to the command line and move to a directory where you want to create a subdirectory with new project. To create a project skeleton enter the command:

mvn archetype:generate

This will run interactive mode for generating new project consisting of following steps:

  1. Enter number corresponding to item “com.google.appengine.archetypes:skeleton-archetype”. It was 56 in my case.
  2. Enter number corresponding to newest available version of Google App Engine. It was 2 in my case corresponding to version 1.7.5.
  3. Enter value for “groupId” of your project
  4. Enter value for “artifactId” of your project – this will be used as a directory name that will be created for your project
  5. Enter value for “version” of your project
  6. Enter name of one of Java packages you’re going to create in your project
  7. Enter “Y” to confirm

Now the project skeleton is ready. Enter the directory with the same name as your “artifactId” and see what’s there. You can remove directory “eclipse-launch-profiles” and files: LICENSE, nbactions.xml, README.md. You need only file “pom.xml” and directory “src”. You should get a similar structure:

Initial files and directories

Configure project dependecies

Adjust version of Google App Engine

We would like to use ver. 1.9.0 of GAE instead of 1.7.5 that was provided by Maven, so open file pom.xml and change the tag <appengine.target.version> to:

<appengine.target.version>1.9.0</appengine.target.version>

JPA 2.0

By default Google App Engine is supporting JPA 1.0 as one of Java persistence technologies that allows to avoid tightly coupling with Google App Engine storage (some kind of no-SQL database). To use JPA 2.0 we need:

  1. JPA 2.0 API dependency
  2. Datanucleus provider of JPA 2.0 in version corresponding to our version of Google App Engine SDK
  3. Datanucleus for Google App Engine in version corresponding to our version of Google App Engine SDK
  4. Configure Datanucleus plugin for Maven that will process entity classes enhancement, as required by Datanucleus JPA implementation

persistence.xml

There are 2 important things related to persistence.xml file:

  • proper location of the file in the project, which should be:
    src/main/resources/META-INF/persistence.xml
    I checked that this location works though GAE documentation gives different location
  • name of persistence-unit (“appengine-persistence-unit” in my example), which we’re going to refer from pom.xml – the name itself is arbitrarily chosen

The skeleton of persistence.xml file in my approach is following:

<?xml version="1.0" encoding="UTF-8"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence"
			 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
			 xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://www.oracle.com/webfolder/technetwork/jsc/xml/ns/persistence/persistence_2_0.xsd"
			 version="2.0">

	<persistence-unit name="appengine-persistence-unit">
<!-- You will list your entity classes here:
		<class>kt.samples.SomeEntity</class>
-->
		<exclude-unlisted-classes/>
		<provider>org.datanucleus.api.jpa.PersistenceProviderImpl</provider>
		<properties>
			<property name="datanucleus.NontransactionalRead" value="true"/>
			<property name="datanucleus.NontransactionalWrite" value="true"/>
			<property name="datanucleus.ConnectionURL" value="appengine"/>
			<property name="datanucleus.jpa.addClassTransformer" value="false"/>
			<property name="datanucleus.appengine.datastoreEnableXGTransactions" value="true"/>
		</properties>
	</persistence-unit>
</persistence>

Please note following:

  • Contrary to example from GAE documentation I referred XML schema for JPA 2.0 and denoted it in “version” parameter of tag <persistence>
  • I chose to list entity classes what allows later to precisely control which classes will be processed by Datanuclues enhancer
  • I set “datanucleus.jpa.addClassTransformer” property to “false” what is important as classes in our project will be enhanced during build time. Without this there was a runtime error.
  • I set “datanucleus.appengine.datastoreEnableXGTransactions” property to “true”, so we’re be less limited by Google App Engine storage specific features. Without this a single transaction can access only entities belonging to a single, so called, “entity group”.

Other properties are standard for Google App Engine and JPA 2.

pom.xml

Let’s define version numbers for Datanucleus implementation of JPA 2 and Datanucleus for GAE by adding below code inside <properties> tag in pom.xml:

<datanucleus.jpa.version>3.1.3</datanucleus.jpa.version>
<datanuclues.appengine.ver>2.1.2</datanuclues.appengine.ver>

These version numbers are established by downloading Google App Engine SDK for Java and examining a content of directory lib/opt/user/datanucleus/v2 in the archive.

Next add following code inside tag <dependencies> in pom.xml:

<dependency>
	<groupId>org.eclipse.persistence</groupId>
	<artifactId>javax.persistence</artifactId>
	<version>2.0.5</version>
</dependency>		
<dependency>
	<groupId>org.datanucleus</groupId>
	<artifactId>datanucleus-core</artifactId>
	<version>${datanucleus.jpa.version}</version>
	<scope>runtime</scope>
</dependency>
<dependency>
	<groupId>org.datanucleus</groupId>
	<artifactId>datanucleus-api-jpa</artifactId>
	<version>${datanucleus.jpa.version}</version>
</dependency>
<dependency>  
	<groupId>com.google.appengine.orm</groupId>  
	<artifactId>datanucleus-appengine</artifactId>  
	<version>${datanuclues.appengine.ver}</version>  
</dependency>

I used JPA 2 API library from org.eclipse.persistence instead of library from org.apache.geronimo.specs by my arbitrary choice. However GAE SDK provides the latter one.

And finally we set up a Maven plugin performing Datanucleus enhancement of JPA entity classes by inserting following code into content of <plugins> tag in pom.xml (I have inserted it as a second plugin):

<plugin>  
	<groupId>org.datanucleus</groupId>  
	<artifactId>maven-datanucleus-plugin</artifactId>  
	<version>${datanucleus.jpa.version}</version>  
    
	<configuration>  
		<api>JPA</api>
		<verbose>true</verbose>
		<persistenceUnitName>appengine-persistence-unit</persistenceUnitName> 
	</configuration>  
  
	<dependencies>  
		<dependency>  
			<groupId>org.datanucleus</groupId>  
			<artifactId>datanucleus-core</artifactId>  
			<version>${datanucleus.jpa.version}</version>  
		</dependency>  
		<dependency>
			<groupId>org.datanucleus</groupId>
			<artifactId>datanucleus-api-jpa</artifactId>
			<version>${datanucleus.jpa.version}</version>
		</dependency>					
	</dependencies>
    
	<executions>
		<execution>
			<phase>compile</phase>
			<goals>
				<goal>enhance</goal>
			</goals>
		</execution>
	</executions>  
</plugin> 

Please note that the plugin refers to our persistence unit name thus the plugin will only enhance entity classes pointed by the persistence unit definition.

Spring Framework

From Spring Framework we need:

  1. spring-context – needed for Spring beans configuration, especially based on annotations
  2. spring-web – needed for Spring beans context initialization in web application
  3. spring-tx – needed for Spring based database transactions
  4. spring-orm – needed for creation of beans related to JPA like entity manager factory

So let’s define a common version number for Spring libraries adding below code to <properties> tag of pom.xml:

<spring.version>3.2.8.RELEASE</spring.version>

And let’s insert following code to content of <dependencies> tag in pom.xml:

<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-context</artifactId>
	<version>${spring.version}</version>
</dependency>
<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-web</artifactId>
	<version>${spring.version}</version>
</dependency>
<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-tx</artifactId>
	<version>${spring.version}</version>
</dependency>            
<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-orm</artifactId>
	<version>${spring.version}</version>
</dependency>

Write code!

Let’s say in our example application we would like to store foreign currency exchange rates. Each record consists of: 1st currency, 2nd currency, exchange rate from 1st currency to 2nd currency. Each day we would like to store plenty of these records (for each pair of currencies there will be 2 exchange rates: from the 1st currency to the 2nd and from the 2nd to the 1st – they’re aren’t equivalent).

Entity classes

In Google App Engine storage system entities are related in sense of parent-child relation. Parent entity must be stored first. All entities having the same ancestor belong to the same entity group. Plain transactions can work only on entities from a single group. Cross-group (XG) transactions can work on up to 5 entity groups. That’s why I set “datanucleus.appengine.datastoreEnableXGTransactions” property to “true” in persistence.xml.

If we design only a single entity, than we will be limited to store only 5 exchange rates in a single transaction. The Google App Engine specific features forces us to introduce a parent entity. At beginning it looks weird. But often you can find some natural parent entity. And sometimes not. Here is an example of such root-entity (parent entity):

@Entity
public class FxSources {
	
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	private Long id;
	
	@Column(nullable = false)	
	@Temporal(TemporalType.DATE)
	private Date createdDate;
	
	@OneToMany(cascade = CascadeType.ALL, mappedBy = "parent")	
	private List<FxSource> sources;
	
	public Long getId() {
		return id;
	}

	public Date getCreatedDate() {
		return createdDate;
	}

	public List<FxSource> getSources() {
		if (sources == null) {
			sources = new ArrayList<FxSource>();
		}
		return sources;
	}

	public void setCreatedDate(Date createdDate) {
		this.createdDate = createdDate;
	}
}

And below is an example of a domain entity (child entity). You want to operate with many such entities in a single transaction, so it must be child of some other entity.

@Entity
public class FxSource {
	
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	@Extension(key="gae.encoded-pk", value="true", vendorName="datanucleus")
	private String id;
	
	@Column(nullable = false)
	private String description;
	
	@ManyToOne(fetch = FetchType.LAZY, optional = false)
	private FxSources parent;
	
	public FxSource() {
	}
	
	public FxSource(String description, FxSources parent) {
		this.description = description;
		this.parent = parent;
	}
	
	public String getId() {
		return id;
	}

	public String getDescription() {
		return description;
	}

	public void setDescription(String description) {
		this.description = description;
	}
	
	@Override
	public String toString() {
		if (id == null) {
			return description;
		}
		else {
			return description + " [id=" + id + ']';
		}
	}
}

Remember this solution will limit you to operate only at most on 5 root-entities at once. So it can happen you will need to introduce another root-entity that will be parent of the entity that was going to be root-entity at the beginning. In general, for Google App Engine and JPA, you have to design your root parent entity such a way there will be at most couple of such records in your problem domain.

IMPORTANT: it looks from GAE documentation that if one wants to have custom primary keys (not auto generated primary keys) then the only option is to use String as a type for @Id field. At first approach I used Integer type for field year but the code failed on searching (EntityManager.find). Switching to String solved the problem.

Below is another entity class acting as a child of one entity (FxDay) and as parent for other entity (FxRate).

@Entity
public class FxRatesPack {
	
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	@Extension(key="gae.encoded-pk", value="true", vendorName="datanucleus")
	private String id;	
	
	@Column(nullable = false)
	private String sourceId;
	
	/**
	 * Number representing time. Format: HHMMSS
	 */
	@Column(nullable = false, length = 8)
	private int downloadTime;
	
	@ManyToOne(fetch = FetchType.LAZY, optional = false)
	private FxDay day;
	
	@OneToMany(cascade = CascadeType.ALL, mappedBy = "pack")
	private Set<FxRate> fxRates = new TreeSet<FxRate>();

	
	public String getId() {
		return id;
	}

	public String getSourceId() {
		return sourceId;
	}

	public void setSourceId(String sourceId) {
		this.sourceId = sourceId;
	}

	public int getDownloadTime() {
		return downloadTime;
	}

	public void setDownloadTime(int time) {
		this.downloadTime = time;
	}

	public Set<FxRate> getFxRates() {
		return fxRates;
	}

	public void setFxRates(Set<FxRate> fxRates) {
		this.fxRates = fxRates;
	}

	public FxDay getDay() {
		return day;
	}

	public void setDay(FxDay day) {
		this.day = day;
		day.getFxRatesPacks().add(this);
	}
}

And below is our main entity class:

@Entity
public class FxRate implements Comparable<FxRate> {
	
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	@Extension(key="gae.encoded-pk", value="true", vendorName="datanucleus")
	private String id;	
	
	@Column(nullable = false)
	private CcyCode fromCcy;
	
	@Column(nullable = false)
	private CcyCode toCcy;
	
	@Column(nullable = false)
	private FxType type;
	
	@Column(nullable = false)
	private BigDecimal fx;
	
	@ManyToOne(fetch = FetchType.LAZY, optional = false)
	private FxRatesPack pack;
	
	public FxRate() {
	}
	
	public FxRate(CcyCode fromCcy, CcyCode toCcy, FxType type, BigDecimal fx) {
		if(fromCcy == null || toCcy == null || type == null || fx == null) {
			throw new IllegalArgumentException("All FX data must be non-null");
		}
		this.fromCcy = fromCcy;
		this.toCcy = toCcy;
		this.type = type;
		this.fx = fx;
	}
	
	public CcyCode getFromCcy() {
		return fromCcy;
	}
	public CcyCode getToCcy() {
		return toCcy;
	}
	public BigDecimal getFx() {
		return fx;
	}
	public void setFx(BigDecimal rate) {
		fx = rate;
	}

	public FxType getType() {
		return type;
	}

	public FxRatesPack getPack() {
		return pack;
	}

	public void setPack(FxRatesPack pack) {
		this.pack = pack;
		pack.getFxRates().add(this);
	}

	@Override
	public String toString() {
		return fromCcy + " -> " + toCcy + ": " + ((type == FxType.INVERTED) ? "1/" : "") + fx;
	}
	
	@Override
	public boolean equals(Object o) {
		if(o instanceof FxRate) {
			FxRate x = (FxRate)o;
			return fromCcy.equals(x.fromCcy) && toCcy.equals(x.toCcy) && type == x.type;
		}
		else {
			return false;
		}
	}

	@Override
	public int hashCode() {
		return fromCcy.hashCode() + toCcy.hashCode() * 13 + type.hashCode() * 17;
	}

	@Override
	public int compareTo(FxRate x) {
		if(x == null) {
			return 1;
		}
		
		int compResult = fromCcy.compareTo(x.fromCcy);
		if(compResult != 0) {
			return compResult;
		}
		
		compResult = toCcy.compareTo(x.toCcy);
		if(compResult != 0) {
			return compResult;
		}
		
		compResult = type.compareTo(x.type);
		if(compResult != 0) {
			return compResult;
		}		
		
		return 0;
	}
}

IMPORTANT: a child entity class for JPA in Google App Engine must have a special primary key. One of available options is to use field of type String annotated with JPA annotations for primary key and one GAE-specific annotation: @org.datanucleus.api.jpa.annotations.Extension(key=”gae.encoded-pk”, value=”true”, vendorName=”datanucleus”)

Now, let’s complete the persistence.xml file by listing all entity classes (class name with package) inside <persistence-unit> tag. Only classes listed will be enhanced by the Datanucleus plugin.

Spring configuration

Let’s create applicationContext.xml file in src/main/webapp/WEB-INF directory. The content of the file:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	   xmlns:context="http://www.springframework.org/schema/context"
	   xmlns:tx="http://www.springframework.org/schema/tx"
	   xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
							http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
							http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd">
	
	<context:annotation-config/>
	<context:component-scan base-package="kt.samples"/>
		
	<tx:annotation-driven transaction-manager="transactionManager"/>
	
	<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
		<constructor-arg ref="entityManagerFactory"/>
	</bean>
	
	<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
		<property name="persistenceUnitName" value="appengine-persistence-unit"/>
	</bean>
	
	<bean class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor"/>
		
</beans>

Thanks to this configuration we will have:
– all annotated beans in package kt.samples detected and initialized in Spring context
– JPA entity manager support by injecting EntityManager with @PersistenceContext annotation in a Spring bean
– transactions support based on Spring annotations
– database exceptions translation to Spring provided classes from DataAccessException hierarchy

Next let’s setup creation of Spring application context like in a normal web application. For this edit src/main/webapp/WEB-INF/web.xml file and insert below code inside <web-app> tags:

<context-param>
    <param-name>contextConfigLocation</param-name>
    <param-value>/WEB-INF/applicationContext.xml</param-value>
</context-param>
<listener>
    <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
</listener>

Repository class sample

@Repository
public class FxRepository {
	
	@PersistenceContext
	EntityManager entityManager;
	
	private FxSources findOrCreateFxSourceRoot(Date timestamp) {
		
		TypedQuery<FxSources> query = entityManager.createQuery(
				"select s from " + FxSources.class.getName() + " s", FxSources.class);
		List<FxSources> result = query.getResultList();
		
		FxSources fxSources;
		if (result.isEmpty()) {
			fxSources = new FxSources();
			fxSources.setCreatedDate(timestamp);
			entityManager.persist(fxSources);
			entityManager.flush();
			entityManager.refresh(fxSources);
		}
		else {
			fxSources = result.get(0);
		}
		return fxSources;
	}
	
	@Transactional
	public void store(Set<FxRate> fxRates, String sourceDesc, Date fxDate) {
		
		FxSource existingFxSrc = findFxSourceByDesc(sourceDesc);
		if (existingFxSrc == null) {
			
			FxSources parent = findOrCreateFxSourceRoot(fxDate);
			existingFxSrc = new FxSource(sourceDesc, parent);
			parent.getSources().add(existingFxSrc);
			
			entityManager.persist(existingFxSrc);
			entityManager.flush();
			entityManager.refresh(existingFxSrc);
		}
		
		// GAE specific: "parent" entity must be persisted first...
		final String dayDate = DateUtil.formatDate(fxDate);
		
		FxDay existingDay = findFxDay(dayDate);
		if (existingDay == null) {
			existingDay = new FxDay();
			existingDay.setDayDate(dayDate);
			entityManager.persist(existingDay);
			entityManager.flush();
			entityManager.refresh(existingDay);
		}
		
		final int fxDateAsInt = DateUtil.timeAsInt(fxDate);
		
		// check if data were stored already
		if (!isFxRatesPackStored(existingFxSrc, fxDateAsInt)) {
		
			// GAE specific: ...then child entities are persisted
			FxRatesPack pack = new FxRatesPack();
			pack.setDownloadTime(fxDateAsInt);
			pack.setSourceId(existingFxSrc.getId());
			pack.setDay(existingDay);

			for (FxRate fxRate : fxRates) {
				fxRate.setPack(pack);
			}
		}
	}
	
	public FxSource findFxSourceByDesc(String desc) {
		TypedQuery<FxSource> query = entityManager.createQuery(
				"select s from " + FxSource.class.getName() + " s where s.description = :desc",
				FxSource.class);
		query.setParameter("desc", desc);
		return JpaUtil.singleResult(query, "description", desc);
	}
	
	public List<FxRatesPack> getAllFxRatesPacks() {
	
		TypedQuery<FxRatesPack> query = entityManager.createQuery(
				"select p from " + FxRatesPack.class.getName() + " p order by p.downloadTime, p.sourceId",
				FxRatesPack.class);
		return query.getResultList();
	}
	
	public FxDay findFxDay(String dayDate) {
		return entityManager.find(FxDay.class, dayDate);
	}

	public List<FxSource> getFxSources(Set<String> sourceIds) {
		
		StringBuilder queryTxt = new StringBuilder();
		queryTxt.append("select s from ");
		queryTxt.append(FxSource.class.getName());
		queryTxt.append(" s where");
		appendPlaceholders(queryTxt, sourceIds.size());

		TypedQuery<FxSource> query = entityManager.createQuery(queryTxt.toString(), FxSource.class);
		Iterator<String> iter = sourceIds.iterator();
		int i = 1;
		while (iter.hasNext()) {
			query.setParameter(i, iter.next());
			++i;
		}
		
		return query.getResultList();
	}
	
	private static void appendPlaceholders(StringBuilder queryTxt, int param_count) {
		for (int i = 0; i < param_count; ++i) {
			if (i > 0) {
				queryTxt.append(" or");
			}
			queryTxt.append(" s.id = ?").append(i + 1);
		}
	}

	private boolean isFxRatesPackStored(FxSource existingFxSrc, int fxDateAsInt) {

		TypedQuery<FxRatesPack> query = entityManager.createQuery(
				"select p from " + FxRatesPack.class.getName() + " p where p.downloadTime = :time and p.sourceId = :srcId",
				FxRatesPack.class);
		query.setParameter("time", fxDateAsInt);
		query.setParameter("srcId", existingFxSrc.getId());
		return query.getResultList().isEmpty() == false;
	}
}


(here the draft ends, so that’s all folks)

Posted in GoogleAppEngine, Java, Spring | Tagged , , | 1 Comment

RSA signatures in Java with Bouncy Castle

This is a complete guide, starting from RSA key pair generation into PEM files, through loading private/public keys from files into proper Bouncy Castle objects, to digital signature creation and verification – all using Bouncy Castle. No JCA.

How to generate RSA private and public keys ready for Java?

I know 3 programs that can generate RSA keys:

The first tool is intended only for Java world thus it is less universal. Moreover it’s harder with keytool to prepare a public key and private key in separate files. And I really don’t like to consider if there are any RSA key size limits in JDK. So I resigned from it.

After some research I think one should use openssl instead of ssh-keygen. The first one allows to write files with public and private keys in a format that can be loaded by a Java program. And it generates as good keys as the second one as both tools are based on Open SSL library.

Using openssl

Below bash commands lead to obtain 2 separate files with 4096-bit RSA key pair: one file with private key and one file with public key. Both files in PEM format which is a plain text. I’ve decided to use PEM format as it’s quite convenient and universal.

openssl genrsa -out priv-key.pem 4096
openssl rsa -in priv-key.pem -pubout -outform PEM -out pub-key.pem

After executing above commands we have an RSA public key in pub-key.pem file and the related RSA private key in priv-key.pem file.

How to create and check RSA signature in Java?

We will use Bouncy Castle cryptographic library. My experience is that JCA can be dissapointing (example: key strengh limits). To use Bouncy Castle we need these dependencies (Maven):

        <dependency>
            <groupId>org.bouncycastle</groupId>
            <artifactId>bcprov-jdk15on</artifactId>
            <version>1.56</version>
        </dependency>
        <dependency>
            <groupId>org.bouncycastle</groupId>
            <artifactId>bcpkix-jdk15on</artifactId>
            <version>1.56</version>
        </dependency>

The second dependency is for class org.bouncycastle.openssl.PEMParser which we will use to load private and public keys from generated PEM files.

Loading RSA keys from PEM files

First we need to be able to load RSA private or public key from a disk file into a Java object of a proper class from Bouncy Castle. For this task I propose a following Java code:

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import org.apache.commons.lang3.Validate;
import org.bouncycastle.asn1.pkcs.PrivateKeyInfo;
import org.bouncycastle.asn1.x509.SubjectPublicKeyInfo;
import org.bouncycastle.crypto.params.AsymmetricKeyParameter;
import org.bouncycastle.crypto.util.PrivateKeyFactory;
import org.bouncycastle.crypto.util.PublicKeyFactory;
import org.bouncycastle.openssl.PEMKeyPair;
import org.bouncycastle.openssl.PEMParser;

public class KeyUtil {
	
	public static AsymmetricKeyParameter loadPublicKey(InputStream is) {
		SubjectPublicKeyInfo spki = (SubjectPublicKeyInfo) readPemObject(is);
		try {
			return PublicKeyFactory.createKey(spki);
		} catch (IOException ex) {
			throw new RuntimeException("Cannot create public key object based on input data", ex);
		}
	}
	
	public static AsymmetricKeyParameter loadPrivateKey(InputStream is) {
		PEMKeyPair keyPair = (PEMKeyPair) readPemObject(is);
		PrivateKeyInfo pki = keyPair.getPrivateKeyInfo();
		try {
			return PrivateKeyFactory.createKey(pki);
		} catch (IOException ex) {
			throw new RuntimeException("Cannot create private key object based on input data", ex);
		}
	}

	private static Object readPemObject(InputStream is) {
		try {
			Validate.notNull(is, "Input data stream cannot be null");
			InputStreamReader isr = new InputStreamReader(is, "UTF-8");
			PEMParser pemParser = new PEMParser(isr);
			
			Object obj = pemParser.readObject();
			if (obj == null) {
				throw new Exception("No PEM object found");
			}
			return obj;
		} catch (Throwable ex) {
			throw new RuntimeException("Cannot read PEM object from input data", ex);
		}
	}
}

Actual loading of a private or a public RSA key with use of KeyUtil class looks like this:

// load a public key from the file 'pub-key.pem'
InputStream pubKeyInpStream = new FileInputStream(new File("pub-key.pem"));
AsymmetricKeyParameter publKey = KeyUtil.loadPublicKey(pubKeyInpStream);

// load a private key from the file 'priv-key.pem'
InputStream prvKeyInpStream = new FileInputStream(new File("priv-key.pem"));
AsymmetricKeyParameter privKey = KeyUtil.loadPrivateKey(prvKeyInpStream);

 

RSA digital signature creation and verification

Basically the creation of a digital signature is an encryption of a message digest with use of a private key. The verification consists of decrypting a signature with use of a public key and comparing it to a message digest calculated again from the message.

So one new thing to do is a calculation of a message digest. Here one can choose from couple of different digest algorithms. Bouncy Castle support many of them (please note to choose a digest algorithm that is considered secure). Let’s stick to SHA512 for code samples. This algorithm is provided by org.bouncycastle.crypto.digests.SHA512Digest class.

Another thing to consider is that a “message” may be anything one wants. However before calculating its digest it must be first converted to a byte array. For text messages one can use String.getBytes(Charset). I will just assume our message is in byte array form already.

RSA signing and signature verification is handled by a single class in Bouncy Castle – org.bouncycastle.crypto.signers.RSADigestSigner. Now we have all ingredients!

Creation of an RSA digital signature:

// GIVEN: InputStream prvKeyInpStream
AsymmetricKeyParameter privKey = KeyUtil.loadPrivateKey(prvKeyInpStream);

// GIVEN: byte[] messageBytes = ...
RSADigestSigner signer = new RSADigestSigner(new SHA512Digest());
signer.init(true, privKey);
signer.update(messageBytes, 0, messageBytes.length);

try {
    byte[] signature = signer.generateSignature();
} catch (Exception ex) {
    throw new RuntimeException("Cannot generate RSA signature. " + ex.getMessage(), ex);
}

 

Verification of an RSA digital signature:

Please note that now the first parameter of RSADigestSigner.init(..) method call is false.

// GIVEN: InputStream pubKeyInpStream
AsymmetricKeyParameter publKey = KeyUtil.loadPublicKey(pubKeyInpStream);

// GIVEN: byte[] messageBytes
RSADigestSigner signer = new RSADigestSigner(new SHA512Digest());
signer.init(false, publKey);
signer.update(messageBytes, 0, messageBytes.length);

// GIVEN: byte[] signature - see code sample above
boolean isValidSignature = signer.verifySignature(signature);

 

Here is the real life example of signature verification code: https://github.com/k-tomaszewski/kt-app-monitor/blob/master/src/main/java/kt/appmonitor/RSASignatureVerifier.java

Please leave your comment if you know that something should be improved here to increase signature security.

Posted in Java, security | Tagged , | Leave a comment

Sequence of digits from natural numbers

Around a year ago I was taking a part, as a candidate, in a recruitment process in one of the world’s biggest and most innovative software companies. One question was to propose an algorithm to solve the following problem:

Given an infinite sequence made up of digits (in base of 10) from subsequent natural numbers find an element at a given position.

So the sequence goes like that: 123456789101112131415161718192021… At positions from 1 to 9 we have digits corresponding to numbers from 1 to 9. Next two elements are 1 and 0 because these are digits from number 10 which comes after number 9. And so on.

As I was not happy with an algorithm proposition I’ve presented to my interviewer I’ve decided to work it out. And here it is! And it’s in Java!

Solution description

The naive solution would be to generate the sequence in a loop by concatenating text representations of subsequent natural numbers until the sequence is longer than given position. Then one needs to get the element at the given position.

The complexity of the naive solution is linear: O(n). I wanted something better.

My solution consists of two parts:

  1. Determine (in a loop):
    1. How many digits (D) there are in a natural number that produced a sequence element (digit) at the given position?
    2. How many digits (elements of the sequence) were produced by natural numbers having less then D digits?
  2. Reduce the problem to finding an element of a sequence of digits made up from natural numbers having digits only.
    1. Transform the given position in the initial sequence into a new position into this simplified sequence by subtracting a total number of digits produces by all natural numbers having less than D digits.
    2. Which number having D digits is covering the element at the new position? This gives us an index of the D-digit number. It’s easy as every D elements of the new sequence is corresponding to a single D-digit number.
    3. What this D-digit number is? It’s easy as this is the first D-digit natural number (10 ^ (D – 1)) plus the calculated index.
    4. Which digit of this D-digit number corresponds to the given position? Easy again as this is the remainder of dividing the new position by D. This last step gives the answer to the problem.

Couple of findings:

  • The number of natural numbers having D digits: 9 * 10 ^ (D – 1)
    • There are 9 natural numbers with 1 digit, 90 natural numbers with 2 digits, etc.
  • The number of digits contained in all natural numbers having D digits: D * 9 * 10 ^ (D – 1)
    • All 1-digit natural numbers produce 9 digits, all 2-digits natural numbers produce 180 digits, and so on.

The algorithm in Java

public class WhatDigit {
	
	public static char solve(int position) {
		long n = position;

		// part 1
		int digits = 1;
		long offset = 0;
		long digitsCount = 9;
		while (Math.addExact(offset, digitsCount) < n) {
			offset += digitsCount;
			++digits;
			digitsCount = 9 * digits * BigInteger.TEN.pow(digits - 1).longValueExact();
		}
		
		// part 2
		n -= offset + 1;
		long numberIndex = n / digits;
		long number = BigInteger.TEN.pow(digits - 1).longValueExact() + numberIndex;
		int digitIndex = (int)(n % digits);
		if (digitIndex > 0) {
			number = number % BigInteger.TEN.pow(digits - digitIndex).longValueExact();
		}
		long element = number / BigInteger.TEN.pow(digits - digitIndex - 1).longValueExact();
		
		return (char)(element + '0');
	}
}

Remarks

I believe the complexity is in order of O(log(n)) because:

  1. The first part consists of a loop. The number of iterations is equal to the number of digits in the given position. Thus the number of iterations is proportional to the logarithm of the value of the given position.
  2. The second part could be considered as having a contant-time complexity. However this heavily depends on the complexity of the used implementation of a power function (BigInteger.pow(int) in my case). This is probably a weakness of my solution as the complexity of this implemetation of a power function is rather not O(1)… There is a solution however. As we need only powers of 10 we can iteratively calculate higher and higher powers in the loop in the 1st part using multiplication only and keep them in a dictionary. The highest power of 10 we need is 10 ^ (digits – 1).

In couple of places in calculations based on long values I used methods like Math.addExact(long, long) or BigInteger.longValueExact(). It’s because I wanted overflows on long to be detected and signaled with an exception.

Question at the end: can the part 1 be done in a constant time? 🙂

Nice mind exercise!

Posted in Java | Tagged , | Leave a comment

JavaScript, comparing numbers and null

Last time I’m programming quite a lot in JavaScript. And I discovered this kind of inconsistency when comparing a null value with numbers. Let’s consider this code in JavaScript:

var x = null;
var y = ... // some number

console.info('Comparing null with: ', y);
console.info('lt: ', x < y); 
console.info('gt: ', x > y);
console.info('eq: ', x == y);
console.info('lte: ', x <= y); 
console.info('gte: ', x >= y);

I was assuming null value compared to numbers will behave in a way somehow similar to that like NULL is behaving in SQL. Or at least I was assuming all above comparing expressions will be evaluated to false. The reality is even worse! Results are depending on a number which is value of y variable.

For y = 3 we get:

Comparing null with: 3
lt: true
gt: false
eq: false
lte: true
gte: false

So it looks null is considered in JavaScript to be less than 3.

For y = 0 we get:

Comparing null with: 0
lt: false
gt: false
eq: false
lte: true
gte: true

Total nonsense! null is not less than 0 and is not equal to 0 but… it’s less or equal to 0.

For y = -3 we get:

Comparing null with: -3
lt: false
gt: true
eq: false
lte: false
gte: true

At least we know JavaScript “thinks” null is greater than a negative number! 🙂

The conclusion is that you must be very cautious with null value in JavaScript. It should not be compared with numbers as results are against the logic.

Disclaimer: I was executing JavaScript in console of Firefox ver. 50.

BTW: It looks there are more inconsistencies in JavaScript! 😀

Solution

If you need to compare a variable against numbers in JavaScript and it’s possible this variable can be null then all these comparing expressions should use parseInt(x) instead of x. The value of parseInt(null) is NaN (not-a-number) which behaves nice (in line with common reason) when compared to numbers. So now we have:

console.info('Comparing null with: ', y);
console.info('lt: ', parseInt(x) < y); 
console.info('gt: ', parseInt(x) > y);
console.info('eq: ', parseInt(x) == y);
console.info('lte: ', parseInt(x) <= y); 
console.info('gte: ', parseInt(x) >= y);

Now all these expressions will evaluate to false no matter what number is the value of variable y.

NOTE: If your variable (like x in the example above) is going to hold floating-point numbers then use parseFloat instead of parseInt.

Posted in JavaScript | Leave a comment

Scala like Perl?

These days really many people (in the software development world) are fascinated by Scala. And on the other hand it’s practically impossible to find someone feeling similar about Perl. This is somehow surprising for me as I see similarities between these 2 programming languages. I think one of these similarities is rather unquestionable:

implicit parameters/variables

These are nasty things causing variables coming from nowhere with values silently assigned. Somewhere. In Scala these are called “implicit parameters”. In Perl these are “special variables”.

As far as I know there are no such things in Pascal, C, C++, Java or JavaScript. So you don’t need to study these languages hard to read source of programs written in them. That’s not true in case of Scala or Perl. You need to know these hellish language features to get an idea how the program works. Nice? Cool? Shorter? Unreadable? Expressive? Modern? Shitty?

Well, I don’t like it. And I don’t understand why people do accept one language having such things (Scala) but not the other one (Perl). I’ve found many comments about Scala that it’s modern and fresh. Each time I hear that I think it’s just trendy to use Scala. However “trendy” is something belonging to the world of fashion, not to the world of engineers. On the other hand people often find Perl as programming language hard to read. I guess thanks to these default variables (but maybe I’m wrong).

In that one I would agree: programs written in Scala and in Perl are unreadable (my opinion). And hard to maintain. Or at least it’s really easy to write programs in these languages that way. Such that it’s really unfeasible to fix a bug in such a program for someone not being Scala/Perl fanatic.

Actually I would say Perl has some advantages over Scala. It doesn’t need a JVM. It’s fast. It’s portable. It’s available almost everywhere (in the world of professional IT servers are driven by Unix/Linux which almost always has Perl installed). And Perl has one big disadvantage: it cannot easily use Java frameworks or libraries (what Scala can).

I had to use Scala for integration tests for one project I was working on in 2015. These implicit parameters were my nightmare. So in the huge mass of enthusiastic opinions about Scala I’m really happy there exist others who can take a critical look on it – see the article I Don’t Like Scala by Bozho.

PS. Perl has one funny application of “special variables”. They’re crucial in “Perl golf”!

Posted in Uncategorized | Tagged , , | Leave a comment

Gaming controllers for Linux!

Surprisingly I do sometime play computer games. Of course under Linux! With my sons I like car racing games so I needed a gaming wheel or a gamepad. And here the question arises to which I came as well:

Which gaming wheels or gamepads work well under Linux?

I can recommend 2 of such devices which I’ve checked personally. As I am a casual player please do keep in mind that these are probably not for “professional gamers”. But they’re good enough, have good quality and are not expensive. Both controllers have USB interface. Both are working out of the box under Linux (Linux Mint 17 at least):

  • Genius Speed Wheel 5 Pro – it looks like it can be not available in shops now as I bought it in 2012. These days it costed around 33 USD (105 PLN).
    Genius Speed Wheel 5 Pro

    Genius Speed Wheel 5 Pro

  • Logitech F310 Gamepad – bought at the beginning of 2017 for some less than 20 USD (75 PLN).
    Logitech F310 Gamepad

    Logitech F310 Gamepad

Both controllers are working very well in Grid Autosport under Linux (and it’s the best car racing game I’ve seen on Linux)! On the last Black Friday I’ve bought this game for just 10 EUR! This game is awesome also because it offers split-screen racing mode what I’ve checked with both above-mentioned controllers being in use at the same time!

And I can recommend this application for the initial testing (and some configuration): jstest-gtk

Posted in hardware, Linux | Tagged | Leave a comment