Google App Engine, Java, JPA 2, Spring Framework, Maven

DISCLAIMER: This was going (in 2014!) to be a full tutorial but at some point I’ve lost motivation in digging in the Google App Engine. Actually I was so tired with this GAE data store that I felt really relieved when returned to using a classic relational database. The draft of this article was hanging for couple of years waiting to be completed. Now I’ve finally realized it will never be. As far as I recall the main problem with Google App Engine data store was related to using transactions. Probably things behave a little bit better if you… are not using transations. So be warned:

  • I publish this as a DRAFT just in case someone will find it usefull.
  • The state of knowledge about GAE here is somewhere in 2014

This tries to present how to create a Maven managed project of a Java web application to be deployed on Google App Engine and making use of Spring Framework and JPA 2 as a persistence layer. It shows how to store related entities (@OneToMany) in a transaction (key complication factor) and how to mark JPA entity as a child entity in sense of Google App Engine storage. The tutorial assumes some basic knowledge of Java, Maven, Spring Framework and JPA. I used following software versions:

  1. Google App Engine SDK, ver. 1.9.0
  2. Java SDK ver. 1.7, as required by the above version of Google App Engine
  3. JPA 2.0
  4. Spring Framework ver. 3.2.8
  5. Maven ver. 3.2.1

Create a project skeleton

Go to the command line and move to a directory where you want to create a subdirectory with new project. To create a project skeleton enter the command:

mvn archetype:generate

This will run interactive mode for generating new project consisting of following steps:

  1. Enter number corresponding to item “com.google.appengine.archetypes:skeleton-archetype”. It was 56 in my case.
  2. Enter number corresponding to newest available version of Google App Engine. It was 2 in my case corresponding to version 1.7.5.
  3. Enter value for “groupId” of your project
  4. Enter value for “artifactId” of your project – this will be used as a directory name that will be created for your project
  5. Enter value for “version” of your project
  6. Enter name of one of Java packages you’re going to create in your project
  7. Enter “Y” to confirm

Now the project skeleton is ready. Enter the directory with the same name as your “artifactId” and see what’s there. You can remove directory “eclipse-launch-profiles” and files: LICENSE, nbactions.xml, README.md. You need only file “pom.xml” and directory “src”. You should get a similar structure:

Initial files and directories

Configure project dependecies

Adjust version of Google App Engine

We would like to use ver. 1.9.0 of GAE instead of 1.7.5 that was provided by Maven, so open file pom.xml and change the tag <appengine.target.version> to:

<appengine.target.version>1.9.0</appengine.target.version>

JPA 2.0

By default Google App Engine is supporting JPA 1.0 as one of Java persistence technologies that allows to avoid tightly coupling with Google App Engine storage (some kind of no-SQL database). To use JPA 2.0 we need:

  1. JPA 2.0 API dependency
  2. Datanucleus provider of JPA 2.0 in version corresponding to our version of Google App Engine SDK
  3. Datanucleus for Google App Engine in version corresponding to our version of Google App Engine SDK
  4. Configure Datanucleus plugin for Maven that will process entity classes enhancement, as required by Datanucleus JPA implementation

persistence.xml

There are 2 important things related to persistence.xml file:

  • proper location of the file in the project, which should be:
    src/main/resources/META-INF/persistence.xml
    I checked that this location works though GAE documentation gives different location
  • name of persistence-unit (“appengine-persistence-unit” in my example), which we’re going to refer from pom.xml – the name itself is arbitrarily chosen

The skeleton of persistence.xml file in my approach is following:

<?xml version="1.0" encoding="UTF-8"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence"
			 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
			 xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://www.oracle.com/webfolder/technetwork/jsc/xml/ns/persistence/persistence_2_0.xsd"
			 version="2.0">

	<persistence-unit name="appengine-persistence-unit">
<!-- You will list your entity classes here:
		<class>kt.samples.SomeEntity</class>
-->
		<exclude-unlisted-classes/>
		<provider>org.datanucleus.api.jpa.PersistenceProviderImpl</provider>
		<properties>
			<property name="datanucleus.NontransactionalRead" value="true"/>
			<property name="datanucleus.NontransactionalWrite" value="true"/>
			<property name="datanucleus.ConnectionURL" value="appengine"/>
			<property name="datanucleus.jpa.addClassTransformer" value="false"/>
			<property name="datanucleus.appengine.datastoreEnableXGTransactions" value="true"/>
		</properties>
	</persistence-unit>
</persistence>

Please note following:

  • Contrary to example from GAE documentation I referred XML schema for JPA 2.0 and denoted it in “version” parameter of tag <persistence>
  • I chose to list entity classes what allows later to precisely control which classes will be processed by Datanuclues enhancer
  • I set “datanucleus.jpa.addClassTransformer” property to “false” what is important as classes in our project will be enhanced during build time. Without this there was a runtime error.
  • I set “datanucleus.appengine.datastoreEnableXGTransactions” property to “true”, so we’re be less limited by Google App Engine storage specific features. Without this a single transaction can access only entities belonging to a single, so called, “entity group”.

Other properties are standard for Google App Engine and JPA 2.

pom.xml

Let’s define version numbers for Datanucleus implementation of JPA 2 and Datanucleus for GAE by adding below code inside <properties> tag in pom.xml:

<datanucleus.jpa.version>3.1.3</datanucleus.jpa.version>
<datanuclues.appengine.ver>2.1.2</datanuclues.appengine.ver>

These version numbers are established by downloading Google App Engine SDK for Java and examining a content of directory lib/opt/user/datanucleus/v2 in the archive.

Next add following code inside tag <dependencies> in pom.xml:

<dependency>
	<groupId>org.eclipse.persistence</groupId>
	<artifactId>javax.persistence</artifactId>
	<version>2.0.5</version>
</dependency>		
<dependency>
	<groupId>org.datanucleus</groupId>
	<artifactId>datanucleus-core</artifactId>
	<version>${datanucleus.jpa.version}</version>
	<scope>runtime</scope>
</dependency>
<dependency>
	<groupId>org.datanucleus</groupId>
	<artifactId>datanucleus-api-jpa</artifactId>
	<version>${datanucleus.jpa.version}</version>
</dependency>
<dependency>  
	<groupId>com.google.appengine.orm</groupId>  
	<artifactId>datanucleus-appengine</artifactId>  
	<version>${datanuclues.appengine.ver}</version>  
</dependency>

I used JPA 2 API library from org.eclipse.persistence instead of library from org.apache.geronimo.specs by my arbitrary choice. However GAE SDK provides the latter one.

And finally we set up a Maven plugin performing Datanucleus enhancement of JPA entity classes by inserting following code into content of <plugins> tag in pom.xml (I have inserted it as a second plugin):

<plugin>  
	<groupId>org.datanucleus</groupId>  
	<artifactId>maven-datanucleus-plugin</artifactId>  
	<version>${datanucleus.jpa.version}</version>  
    
	<configuration>  
		<api>JPA</api>
		<verbose>true</verbose>
		<persistenceUnitName>appengine-persistence-unit</persistenceUnitName> 
	</configuration>  
  
	<dependencies>  
		<dependency>  
			<groupId>org.datanucleus</groupId>  
			<artifactId>datanucleus-core</artifactId>  
			<version>${datanucleus.jpa.version}</version>  
		</dependency>  
		<dependency>
			<groupId>org.datanucleus</groupId>
			<artifactId>datanucleus-api-jpa</artifactId>
			<version>${datanucleus.jpa.version}</version>
		</dependency>					
	</dependencies>
    
	<executions>
		<execution>
			<phase>compile</phase>
			<goals>
				<goal>enhance</goal>
			</goals>
		</execution>
	</executions>  
</plugin> 

Please note that the plugin refers to our persistence unit name thus the plugin will only enhance entity classes pointed by the persistence unit definition.

Spring Framework

From Spring Framework we need:

  1. spring-context – needed for Spring beans configuration, especially based on annotations
  2. spring-web – needed for Spring beans context initialization in web application
  3. spring-tx – needed for Spring based database transactions
  4. spring-orm – needed for creation of beans related to JPA like entity manager factory

So let’s define a common version number for Spring libraries adding below code to <properties> tag of pom.xml:

<spring.version>3.2.8.RELEASE</spring.version>

And let’s insert following code to content of <dependencies> tag in pom.xml:

<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-context</artifactId>
	<version>${spring.version}</version>
</dependency>
<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-web</artifactId>
	<version>${spring.version}</version>
</dependency>
<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-tx</artifactId>
	<version>${spring.version}</version>
</dependency>            
<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-orm</artifactId>
	<version>${spring.version}</version>
</dependency>

Write code!

Let’s say in our example application we would like to store foreign currency exchange rates. Each record consists of: 1st currency, 2nd currency, exchange rate from 1st currency to 2nd currency. Each day we would like to store plenty of these records (for each pair of currencies there will be 2 exchange rates: from the 1st currency to the 2nd and from the 2nd to the 1st – they’re aren’t equivalent).

Entity classes

In Google App Engine storage system entities are related in sense of parent-child relation. Parent entity must be stored first. All entities having the same ancestor belong to the same entity group. Plain transactions can work only on entities from a single group. Cross-group (XG) transactions can work on up to 5 entity groups. That’s why I set “datanucleus.appengine.datastoreEnableXGTransactions” property to “true” in persistence.xml.

If we design only a single entity, than we will be limited to store only 5 exchange rates in a single transaction. The Google App Engine specific features forces us to introduce a parent entity. At beginning it looks weird. But often you can find some natural parent entity. And sometimes not. Here is an example of such root-entity (parent entity):

@Entity
public class FxSources {
	
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	private Long id;
	
	@Column(nullable = false)	
	@Temporal(TemporalType.DATE)
	private Date createdDate;
	
	@OneToMany(cascade = CascadeType.ALL, mappedBy = "parent")	
	private List<FxSource> sources;
	
	public Long getId() {
		return id;
	}

	public Date getCreatedDate() {
		return createdDate;
	}

	public List<FxSource> getSources() {
		if (sources == null) {
			sources = new ArrayList<FxSource>();
		}
		return sources;
	}

	public void setCreatedDate(Date createdDate) {
		this.createdDate = createdDate;
	}
}

And below is an example of a domain entity (child entity). You want to operate with many such entities in a single transaction, so it must be child of some other entity.

@Entity
public class FxSource {
	
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	@Extension(key="gae.encoded-pk", value="true", vendorName="datanucleus")
	private String id;
	
	@Column(nullable = false)
	private String description;
	
	@ManyToOne(fetch = FetchType.LAZY, optional = false)
	private FxSources parent;
	
	public FxSource() {
	}
	
	public FxSource(String description, FxSources parent) {
		this.description = description;
		this.parent = parent;
	}
	
	public String getId() {
		return id;
	}

	public String getDescription() {
		return description;
	}

	public void setDescription(String description) {
		this.description = description;
	}
	
	@Override
	public String toString() {
		if (id == null) {
			return description;
		}
		else {
			return description + " [id=" + id + ']';
		}
	}
}

Remember this solution will limit you to operate only at most on 5 root-entities at once. So it can happen you will need to introduce another root-entity that will be parent of the entity that was going to be root-entity at the beginning. In general, for Google App Engine and JPA, you have to design your root parent entity such a way there will be at most couple of such records in your problem domain.

IMPORTANT: it looks from GAE documentation that if one wants to have custom primary keys (not auto generated primary keys) then the only option is to use String as a type for @Id field. At first approach I used Integer type for field year but the code failed on searching (EntityManager.find). Switching to String solved the problem.

Below is another entity class acting as a child of one entity (FxDay) and as parent for other entity (FxRate).

@Entity
public class FxRatesPack {
	
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	@Extension(key="gae.encoded-pk", value="true", vendorName="datanucleus")
	private String id;	
	
	@Column(nullable = false)
	private String sourceId;
	
	/**
	 * Number representing time. Format: HHMMSS
	 */
	@Column(nullable = false, length = 8)
	private int downloadTime;
	
	@ManyToOne(fetch = FetchType.LAZY, optional = false)
	private FxDay day;
	
	@OneToMany(cascade = CascadeType.ALL, mappedBy = "pack")
	private Set<FxRate> fxRates = new TreeSet<FxRate>();

	
	public String getId() {
		return id;
	}

	public String getSourceId() {
		return sourceId;
	}

	public void setSourceId(String sourceId) {
		this.sourceId = sourceId;
	}

	public int getDownloadTime() {
		return downloadTime;
	}

	public void setDownloadTime(int time) {
		this.downloadTime = time;
	}

	public Set<FxRate> getFxRates() {
		return fxRates;
	}

	public void setFxRates(Set<FxRate> fxRates) {
		this.fxRates = fxRates;
	}

	public FxDay getDay() {
		return day;
	}

	public void setDay(FxDay day) {
		this.day = day;
		day.getFxRatesPacks().add(this);
	}
}

And below is our main entity class:

@Entity
public class FxRate implements Comparable<FxRate> {
	
	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	@Extension(key="gae.encoded-pk", value="true", vendorName="datanucleus")
	private String id;	
	
	@Column(nullable = false)
	private CcyCode fromCcy;
	
	@Column(nullable = false)
	private CcyCode toCcy;
	
	@Column(nullable = false)
	private FxType type;
	
	@Column(nullable = false)
	private BigDecimal fx;
	
	@ManyToOne(fetch = FetchType.LAZY, optional = false)
	private FxRatesPack pack;
	
	public FxRate() {
	}
	
	public FxRate(CcyCode fromCcy, CcyCode toCcy, FxType type, BigDecimal fx) {
		if(fromCcy == null || toCcy == null || type == null || fx == null) {
			throw new IllegalArgumentException("All FX data must be non-null");
		}
		this.fromCcy = fromCcy;
		this.toCcy = toCcy;
		this.type = type;
		this.fx = fx;
	}
	
	public CcyCode getFromCcy() {
		return fromCcy;
	}
	public CcyCode getToCcy() {
		return toCcy;
	}
	public BigDecimal getFx() {
		return fx;
	}
	public void setFx(BigDecimal rate) {
		fx = rate;
	}

	public FxType getType() {
		return type;
	}

	public FxRatesPack getPack() {
		return pack;
	}

	public void setPack(FxRatesPack pack) {
		this.pack = pack;
		pack.getFxRates().add(this);
	}

	@Override
	public String toString() {
		return fromCcy + " -> " + toCcy + ": " + ((type == FxType.INVERTED) ? "1/" : "") + fx;
	}
	
	@Override
	public boolean equals(Object o) {
		if(o instanceof FxRate) {
			FxRate x = (FxRate)o;
			return fromCcy.equals(x.fromCcy) && toCcy.equals(x.toCcy) && type == x.type;
		}
		else {
			return false;
		}
	}

	@Override
	public int hashCode() {
		return fromCcy.hashCode() + toCcy.hashCode() * 13 + type.hashCode() * 17;
	}

	@Override
	public int compareTo(FxRate x) {
		if(x == null) {
			return 1;
		}
		
		int compResult = fromCcy.compareTo(x.fromCcy);
		if(compResult != 0) {
			return compResult;
		}
		
		compResult = toCcy.compareTo(x.toCcy);
		if(compResult != 0) {
			return compResult;
		}
		
		compResult = type.compareTo(x.type);
		if(compResult != 0) {
			return compResult;
		}		
		
		return 0;
	}
}

IMPORTANT: a child entity class for JPA in Google App Engine must have a special primary key. One of available options is to use field of type String annotated with JPA annotations for primary key and one GAE-specific annotation: @org.datanucleus.api.jpa.annotations.Extension(key=”gae.encoded-pk”, value=”true”, vendorName=”datanucleus”)

Now, let’s complete the persistence.xml file by listing all entity classes (class name with package) inside <persistence-unit> tag. Only classes listed will be enhanced by the Datanucleus plugin.

Spring configuration

Let’s create applicationContext.xml file in src/main/webapp/WEB-INF directory. The content of the file:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	   xmlns:context="http://www.springframework.org/schema/context"
	   xmlns:tx="http://www.springframework.org/schema/tx"
	   xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
							http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
							http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd">
	
	<context:annotation-config/>
	<context:component-scan base-package="kt.samples"/>
		
	<tx:annotation-driven transaction-manager="transactionManager"/>
	
	<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
		<constructor-arg ref="entityManagerFactory"/>
	</bean>
	
	<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
		<property name="persistenceUnitName" value="appengine-persistence-unit"/>
	</bean>
	
	<bean class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor"/>
		
</beans>

Thanks to this configuration we will have:
– all annotated beans in package kt.samples detected and initialized in Spring context
– JPA entity manager support by injecting EntityManager with @PersistenceContext annotation in a Spring bean
– transactions support based on Spring annotations
– database exceptions translation to Spring provided classes from DataAccessException hierarchy

Next let’s setup creation of Spring application context like in a normal web application. For this edit src/main/webapp/WEB-INF/web.xml file and insert below code inside <web-app> tags:

<context-param>
    <param-name>contextConfigLocation</param-name>
    <param-value>/WEB-INF/applicationContext.xml</param-value>
</context-param>
<listener>
    <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
</listener>

Repository class sample

@Repository
public class FxRepository {
	
	@PersistenceContext
	EntityManager entityManager;
	
	private FxSources findOrCreateFxSourceRoot(Date timestamp) {
		
		TypedQuery<FxSources> query = entityManager.createQuery(
				"select s from " + FxSources.class.getName() + " s", FxSources.class);
		List<FxSources> result = query.getResultList();
		
		FxSources fxSources;
		if (result.isEmpty()) {
			fxSources = new FxSources();
			fxSources.setCreatedDate(timestamp);
			entityManager.persist(fxSources);
			entityManager.flush();
			entityManager.refresh(fxSources);
		}
		else {
			fxSources = result.get(0);
		}
		return fxSources;
	}
	
	@Transactional
	public void store(Set<FxRate> fxRates, String sourceDesc, Date fxDate) {
		
		FxSource existingFxSrc = findFxSourceByDesc(sourceDesc);
		if (existingFxSrc == null) {
			
			FxSources parent = findOrCreateFxSourceRoot(fxDate);
			existingFxSrc = new FxSource(sourceDesc, parent);
			parent.getSources().add(existingFxSrc);
			
			entityManager.persist(existingFxSrc);
			entityManager.flush();
			entityManager.refresh(existingFxSrc);
		}
		
		// GAE specific: "parent" entity must be persisted first...
		final String dayDate = DateUtil.formatDate(fxDate);
		
		FxDay existingDay = findFxDay(dayDate);
		if (existingDay == null) {
			existingDay = new FxDay();
			existingDay.setDayDate(dayDate);
			entityManager.persist(existingDay);
			entityManager.flush();
			entityManager.refresh(existingDay);
		}
		
		final int fxDateAsInt = DateUtil.timeAsInt(fxDate);
		
		// check if data were stored already
		if (!isFxRatesPackStored(existingFxSrc, fxDateAsInt)) {
		
			// GAE specific: ...then child entities are persisted
			FxRatesPack pack = new FxRatesPack();
			pack.setDownloadTime(fxDateAsInt);
			pack.setSourceId(existingFxSrc.getId());
			pack.setDay(existingDay);

			for (FxRate fxRate : fxRates) {
				fxRate.setPack(pack);
			}
		}
	}
	
	public FxSource findFxSourceByDesc(String desc) {
		TypedQuery<FxSource> query = entityManager.createQuery(
				"select s from " + FxSource.class.getName() + " s where s.description = :desc",
				FxSource.class);
		query.setParameter("desc", desc);
		return JpaUtil.singleResult(query, "description", desc);
	}
	
	public List<FxRatesPack> getAllFxRatesPacks() {
	
		TypedQuery<FxRatesPack> query = entityManager.createQuery(
				"select p from " + FxRatesPack.class.getName() + " p order by p.downloadTime, p.sourceId",
				FxRatesPack.class);
		return query.getResultList();
	}
	
	public FxDay findFxDay(String dayDate) {
		return entityManager.find(FxDay.class, dayDate);
	}

	public List<FxSource> getFxSources(Set<String> sourceIds) {
		
		StringBuilder queryTxt = new StringBuilder();
		queryTxt.append("select s from ");
		queryTxt.append(FxSource.class.getName());
		queryTxt.append(" s where");
		appendPlaceholders(queryTxt, sourceIds.size());

		TypedQuery<FxSource> query = entityManager.createQuery(queryTxt.toString(), FxSource.class);
		Iterator<String> iter = sourceIds.iterator();
		int i = 1;
		while (iter.hasNext()) {
			query.setParameter(i, iter.next());
			++i;
		}
		
		return query.getResultList();
	}
	
	private static void appendPlaceholders(StringBuilder queryTxt, int param_count) {
		for (int i = 0; i < param_count; ++i) {
			if (i > 0) {
				queryTxt.append(" or");
			}
			queryTxt.append(" s.id = ?").append(i + 1);
		}
	}

	private boolean isFxRatesPackStored(FxSource existingFxSrc, int fxDateAsInt) {

		TypedQuery<FxRatesPack> query = entityManager.createQuery(
				"select p from " + FxRatesPack.class.getName() + " p where p.downloadTime = :time and p.sourceId = :srcId",
				FxRatesPack.class);
		query.setParameter("time", fxDateAsInt);
		query.setParameter("srcId", existingFxSrc.getId());
		return query.getResultList().isEmpty() == false;
	}
}


(here the draft ends, so that’s all folks)

Advertisements

About krzysztoftomaszewski

I've got M.Sc. in software engineering. I graduated in 2005 at Institute of Computer Science, Warsaw University of Technology, Faculty of Electronics and Information Technology. I'm working on computer software design and engineering continuously since 2004.
This entry was posted in GoogleAppEngine, Java, Spring and tagged , , . Bookmark the permalink.

One Response to Google App Engine, Java, JPA 2, Spring Framework, Maven

  1. pln one says:

    +1

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s