Data Manager Library

Data Management utility library which can be used to manage data different type of data contents in applications. Library provides build-in abstraction for creating different type of Data Manager instances based on different data storage solutions and interfaces. E.g upon databases like Postgresql and MongoDB or cloud based storage solutions.

There are two build-in Data Managers available within this library. In-memory and file data manager. They can be activated for the data service through application.properties file. with relaysoft.data.manager.name property. Use either value in-memory or file. In.memory data manager is provided just for testing purposes thus it cannot actually persist the data contents. File Data Manager uses local File IO as persistent storage. In-memory data manager is used by default if no configurations are provided.

Data Managers are controlled with Data Service instances which acts as a gateway to Data Manager for creating new and getting existing Data objects.

Data objects provided by the Data Managers never contains the actual data content itself but the reference ID into separately saved content. Data Manager saves data content into persistent storage depending on manager implementation used by the application or in temporary cache or both. Data content from objects can be read as byte arrays, input streams, text content (string) or as object if Data type supports object handling.

It is not possible to overwrite or append new data content into existing non-empty data objects referenced content. In order to update data content a new data object need to be created with the updated content.

Data Manager supports following data types

  • Byte
    • Generic data type which handles all contents as bytes. Supports any kind of data to be written.
  • Text
    • Same as Byte content, but only indicates that content is in text formatted data and not e.g. binary file.
  • HTML
    • Same as text content, but only indicates that content is in HTML formatted data.
  • Object
    • Generic data type for objects. Any type Serializable object can be written into this data type.
    • All object data types (Object, XML, XMLNS, JSON and YAML) provides getting saved object content directly without the need for parsing it from byte content.
  • XML
    • Object data which supports saving DOM objects.
  • XMLNS
    • Object data which supports saving DOM objects with namespaces.
  • JSON
    • Supports saving JSON objects
    • Uses Jackson data bind library
    • Using JSON data and Jackson is optional feature, which requires adding Jackson dependency on implementing project
  • YAML
    • Supports saving YAML objects
    • Uses Jackson data format library
    • Using YAML data and Jackson is optional feature, which requires adding Jackson dependency on implementing project
  • Composite
    • Special byte data type which enables content overwriting and appending
    • This data type makes exception in the rule for overwriting and appending on existing content
    • Saves other data objects as parts inside the composite data content. Enables replacing existing data parts and adding new parts into part list

Save location (cache/storage) is controlled by persistence type enumeration when creating or updating data. Data Manager can use persistent storage to save data content or just temporary cache or both in which case data content is stored into persistent storage for later use but is read from cache. When persisting it is also possible to compress and/or encrypt the persisted data content.

Data manager provides following persistence types when saving content:

  • CACHED
  • PERSISTENT
  • COMPRESSED
  • ENCRYPTED
  • COMPRESSED_ENCRYPTED

In addition CACHED persistence type is available with all the other persistence types e.g. CACHED_PERSISTENT in which case data content is saved into underlying storage solution and kept in cache. One should always keep data in cache if possible for improved reads into data content.

Data content encryption is controlled through security service properties. By default data manager encrypts the data persistent content. It is also possible to instruct data manager to use unique secret keys for
content encryption in which case the content is encrypted with unique secret key which is then encrypted with given secret.

Alternatively a custom security service implementation can be created and use it with the data manager.

Security service properties:

relaysoft.data.manager.security.active.secret = xxx
relaysoft.data.manager.security.active.ivl = 16
relaysoft.data.manager.security.active.algorithm = AES
relaysoft.data.manager.security.active.keyAlgorithm = SHA-1
relaysoft.data.manager.security.active.transformation = AES/CFB8/NoPadding
relaysoft.data.manager.security.active.keyBytesLength = 128
# If using PBKDF2 algorithms for the key generation, add salt and iteration count into configuration.
relaysoft.data.manager.security.active.salt = xxx
relaysoft.data.manager.security.active.iterations = 16
# If using unique secret keys
relaysoft.data.manager.security.encryptionStrategy = unique
# If using custom security service
relaysoft.data.manager.security.name = class.path.MySecurityService

Furthermore data manager support simple cycling of encrypted data by defining the recessive secrets as inactive new secrets as active into application.properties.

Old secrets:

relaysoft.data.manager.security.inactive.secret = xxx
relaysoft.data.manager.security.inactive.ivl = 16
relaysoft.data.manager.security.inactive.algorithm = AES
relaysoft.data.manager.security.inactive.keyAlgorithm = SHA-1
relaysoft.data.manager.security.inactive.transformation = AES/CFB8/NoPadding

New secrets:

relaysoft.data.manager.security.active.secret = xxx
relaysoft.data.manager.security.active.ivl = 16
relaysoft.data.manager.security.active.algorithm = AES
relaysoft.data.manager.security.active.keyAlgorithm = SHA-1
relaysoft.data.manager.security.active.transformation = AES/CFB8/NoPadding

In order to re-encrypt the data with new secrets:

DataService service = new DefaultDataService();
// Re-encrypt data contents persisted before the given moment.
service.reencryptData(Instant.now());

Data manager caches data objects and their content by using separate cache service. This service is also controlled by cache service properties. Alternatively a custom cache service implementation can be created and use it with the data manager.

Cache service properties:

relaysoft.data.manager.cache.maxElements = 10000
relaysoft.data.manager.cache.scanInterval = 30
relaysoft.data.manager.cache.timeTolive=120
# If using custom cache service
relaysoft.data.manager.cache.name = class.path.MyCacheService

Example:

DataService service = new DefaultDataService();
// Create new data objects with the service
Data data = service.createByteData("Test data content", Charset.forName("UTF-8"), DataPersistenceEnum.CACHED);

Project sources and documentation are available from GitLab.

Maven dependency

<dependency>
    <groupId>net.relaysoft.commons</groupId>
    <artifactId>data-manager</artifactId>
    <version>3.0.0</version>
</dependency>

In case your project is Spring Boot based there is a starter library available for the data manager which auto configures the data service and enables its usage from other Spring beans. It is also easy to create your own data manager bean components which is then automatically bind to data service.

Example:

@Component
public class MyDataManager extends AbstractDataManager implements DataManager{
    //Implementations
}
@Service
public class SomeService {

    @Autowired
    private DataService dataService;

    public void doSomethingWithDataService() {
        ...
        Data data = dataService.createByteData(byte[] array,
        DataPersistenceEnum.CACHED);
        ...
    }

}

Data Manager Spring Boot Starter Maven dependency

<dependency>
    <groupId>net.relaysoft.commons</groupId>
    <artifactId>data-manager-spring-boot-starter</artifactId>
    <version>3.0.0</version>
</dependency>