For demonstration purposes, we're going to build a simple bibliographic resources management tool – BiblioteK. Although BiblioteK aims to be a big name in the industry (we're looking at you, Zotero), but that's a mission for you if you want to help out. For our demo, though, it'll have limited functionality. It'll compress a humble CLI interface to manage and store resources on your computer.
The usability then'll look like this:
# create a resource
btek resources create --kind book --title "Functional Programming in Scala" --authors "Michael Pilquist, Runar Bjarnason, Paul Chiusano" --category Programming --published 2024
# list resources
btek resources list
# Programming
# [8u4xhd] Functional Programming in Scala (Michael Pilquist, Runar Bjarnason, Paul Chiusano, 2024)
# delete
btek resources remove 8u4xhdTo manage material associated with a given resource, let's say a video or a PDF, the user will have to type something similar to this:
# material
btek material add 8u4xhdIf --name 'My PDF' download.pdf
btek material add 8u4xhdIf video.mp4
btek material list
# Programming
# [8u4xhdIf] Functional Programming in Scala
# - [Yif39s3f] My PDF.pdf
# - [123dfsl9] video.mp4
# Deletes only the specified material.
btek material remove Yif39s3f
# Removes the specified material under
btek material remove 8u4xhdIf video.mp4
# Removes all material associated with the resource.
btek material remove 8u4xhdIfZIO leverages the pattern of programming upon interfaces, select implementations for those interfaces, and then build the layers your application needs. We'll explore in decent depth through the post.
For our sample application, we'll have an architecture that resemble something like the following:
flowchart BT
config(App Config)
quill_sqlite("Quill.Sqlite[?]")
quill_sqlite-->config
resources_repo[Resources Repo]
authors_repo[Authors Repo]
categories_repo[Categories Repo]
materials_repo[Materials Repo]
resources_repo-->quill_sqlite
authors_repo-->quill_sqlite
categories_repo-->quill_sqlite
materials_repo-->quill_sqlite
db[(Sqlite DB)]
quill_sqlite-->db
materials_dir[[Materials Directory]]
materials_store[Materials Store]
materials_store-->materials_dir
materials_store-->config
librarian(Librarian)-->materials_store
librarian-->resources_repo
librarian-->materials_repo
librarian-->authors_repo
librarian-->categories_repo
cli([CLI])
cli-->librarian
The models of our application are almost what you'd expect from a simple application, with the difference that I love using the newtype pattern, which consists in having important types like the id of model having its own type that avoids to be automatically conflated with its underlying type. For example, for the Resource.Id type, its underlying type is an UUID, but not any UUID is a Resource's id, and a Resource's id is not any UUID – we'd have convert them manually so is always obvious which is which.
See here what I am talking about:
import monix.newtypes.NewtypeValidated
enum ResourceKind:
case Book
case Video
final case class Resource(
id: Resource.Id,
title: String,
kind: ResourceKind,
category: Option[Category],
publicationYear: Option[Short],
authors: Set[Author],
)
object Resource:
type Id = Id.Type
object Id extends NewtypeValidated[String]:
def apply(value: String): Either[BuildFailure[Type], Type] =
if value.length() == 8 then Right(unsafeCoerce(value))
else Left(BuildFailure("String wasn't 8 characters long"))
def make: UIO[Id] =
makeShortUUID.map: s =>
Id(s).getOrElse(throw new RuntimeException("unreachable"))
end Resource
final case class ResourceRow(
id: Resource.Id,
title: String,
kind: ResourceKind,
publicationYear: Option[Short],
categoryName: Option[Category.Name] = None,
)I'm using the monix's Newtype library, which gives us a simple wrapper around an opaque type, with an apply function to create a newtype from its underlying type, and a newtype#value method to go the other way around. This wrapper can be either a NewtypeWrapper or a NewtypeValidated, the difference being that the later expects an either as its return type for validation.
Here's a glimpse of all our models, which appart from the newtype thing, are quite simple:
final case class Author(id: Author.Id, name: String)
object Author:
type Id = Id.Type
object Id extends NewtypeWrapped[UUID]
final case class Category(name: Category.Name)
object Category:
type Name = Name.Type
object Name extends NewtypeWrapped[String]
final case class Material(id: Material.Id, name: Material.Name, resourceId: Resource.Id)
object Material:
type Id = Id.Type
object Id extends NewtypeWrapped[UUID]
type Name = Name.Type
object Name extends NewtypeWrapped[String]This pattern is not exclusive to ZIO. I've used it with Cats Effect too. It consists in modelling all the errors your application might have into specific case classes with a common inheritance. I think this patter has especial importance because of the +E type in our ZIO monad, because of its covariance, we could operate upon different kinds of error and the compiler will set the error type as their common antecesor, which normally is Throwable. By modelling the errors, we want to get rid of that Throwable and make a DomainError the common antecesor of all error in our app, so we can handle them more gracefully on the edges.
package dev.zio.content.bibliotek.domain
import java.io.IOException
import java.sql.SQLException
import scala.reflect.runtime.universe.*
import scala.util.control.NoStackTrace
sealed trait DomainError(val message: String) extends NoStackTrace
object DomainError:
final case class RepositoryError(exception: SQLException)
extends DomainError(message = exception.getMessage())
final case class IOError(exception: IOException)
extends DomainError(message = exception.getMessage())
case class NotFoundError[A](what: A) extends DomainError(message = s"$what was not found")Our approach to configuration falls a bit away from what's commonly used in applications. Normaly, you'd like to configure your, let's say, http server, from file inside the repository directory, but ours ought to be configured by an user and persisted on their computer. This is why we need a config to know where the config should reside. For this purpose, the user could define a location by setting an environment variable, but, if not set, the default location will be the user's home plus the .bibliotek directory.
Set by the env var
BIBLIOTEK_CONFIG_DIR.
default:$HOME/.bibliotek
For more complex CLI applications, this directory would in turn have a configuration file to parse. For this application, though, configuring the folder is enough, because all we want is a place to put things in there. That's why we're not using something like zio-config, but I beg you to check its documentation for a more robust configuration platform.
Rememeber that a last, all you want to do is to read a file and parse into a case class – not for this one though.
ZIO comes with a first class configuration support. The function ZIO.config[A] either receives a zio.Config as an implicit or explicit parameter. The configuration object can be constructed using zio.Config's static constructors. For example, zio.Config.string("CONFIG_SOURCE") describes that we want a string from a configuration source. Configuration sources for zio.Config can be configured as a runtime layer at application's bootstrap. Here's more information about it. For now, sufices to say that by default the config provider will lookup at environment variables and system properties, which is exactly what we wannna do here.
The directory we're going to use for storage will be either one defined by the user by setting an environment variable, or, in case is not defined, the user.home system property, with a .bibliotek directory appended.
given config: Config[AppConfig] =
Config
.string("BIBLIOTEK_CONFIG_DIR")
.orElse(Config.string("user.home"))
.map(Path(_) / ".bibliotek")
.map(AppConfig.apply)ZIO.config[A] is what brings it into existance, i.e. actually get the values:
def layer = ZLayer.fromZIO(ZIO.config(config))We could just call ZIO.config wherever we want to use our config, but having it as a layer adds flexibility in how we construct our config – using a library like zio-config won't let us have a zio.Config instance right away, but only after some effectful operation.
I remember a talk from the creator of Clojure in which he states that almost every application have to be able to persist data in a database, and this one is no exception.
Creating the application for this post I almost fell into the rabit hole that is file-based storage, using something like JSON to encode it, but even if it seems simplier at first glance, there are a lot of edge cases that you'd have to deal with by yourself and I'm not that smart nor industrius. Hence, I decided to not reinvent the wheel, and use a database system.
As we're going to store everything in the user's computer, it quickly came to my mind SQLite, which let us use our SQL knowledge, while keeping everything in a thight file, and don't need of a running process (like Postgres) to work. The choice is made.
Quill is the official ZIO library for dealing with SQL. It uses a dsl to build queries at compile time. It has a lot of flexibility, but the basic features were enough for building this application.
We'll need to add a dependency on a driver for Sqlite.
"org.xerial" % "sqlite-jdbc" % sqliteVersionOnly one thing fell out of the normal: As we modelled some of our data types using the newtype pattern (see the models section), we have to define custom encoders for them in following sections.
For Quill to know how to access our database, we need a java.sql.DataSource. This source, for Sqlite, could be constructed from the import org.sqlite.SQLiteDataSource, which comes from our driver. With that, and file in which our database will be stored, we can have our instance.
def makeSqliteDataSource(dbFile: Path): UIO[SQLiteDataSource] =
val ds = new SQLiteDataSource()
ZIO.succeed(ds.setUrl(s"jdbc:sqlite:$dbFile")).as(ds)I used ZIO.succeed because of the mutation, although it doesn't really matter in this case.
This function could be used to build our data source from our configuration.
def dataSourceLayer: RLayer[AppConfig, DataSource] =
ZLayer.fromZIO:
for
cfg <- ZIO.service[AppConfig]
filePath = cfg.configDir / "bibliotek.db"
ds <- makeSqliteDataSource(filePath)
_ <- ZIO.attemptBlocking(cfg.configDir.toFile.mkdir()).debug("Created database directory") // (1)
_ <- ZIO.attemptBlocking(filePath.toFile.createNewFile()).debug("Created database file") // (2)
yield dsOnce we have our datasource, I procceed to create the actual directory (1) and file (2) we specified at the datasource. java.io.File#mkdir will only create one level deep the directory, so the user have to specify a directory parent that exists at the configuration.
To bootstrap our database, I thought of a very simple setup function, which loads up an init.sql file from our resources, split the statements using a semicolon (I know is dangerous, but since we know exactly the contents of init.sql, I see no problem) into individual statements, and excecute each using ZIO.foreach, which traverses (this is how this pattern is called) a list and runs an effect on each element, accumulating the results back in a list.
def setupDatabase: RIO[Quill.Sqlite[SnakeCase], Unit] =
for
quill <- ZIO.service[Quill.Sqlite[SnakeCase]] // 1
sql <- ZIO.attemptBlocking(Source.fromResource("init.sql").mkString)
statements = sql.split(";").dropRight(1).map(_ ++ ";")
_ <- ZIO.foreach(statements)(s => quill.executeAction(s)(io.getquill.context.ExecutionInfo.unknown, ()))
yield ()Notice how in (1) we ask for a Quill.Sqlite[SnakeCase] service. Where is coming from? So far, we don't care. ZIO will take care of calling out for it when use this function. This is what the first type parameter of RIO stands for. It's storing the requirements for this function to work, but doesn't make the assumption of where it is comming from.
To get a Quill.Sqlite[SnakeCase] instance, we have our function that returns a DataSource, and that will serve as input for:
val sqliteLayer: RIO[DataSource, Quill.Sqlite[SnakeCase]] = Quill.Sqlite.fromNamingStrategy(SnakeCase)With this, we have our database setup.
This is something I didn't like about Quill, but everything you do have to live within a context – not of a coconut, but of a database system. For this reason, our encoders and decoders have to live in a trait that has a non implemented context variable, which then we need to import its contents.
Let's see the resources schema as an example:
import io.getquill.{MappedEncoding, SnakeCase}
trait ResourceSchema:
given resourceIdEncoder: MappedEncoding[UUID, Resource.Id] = MappedEncoding[UUID, Resource.Id](Resource.Id(_))
given resourceIdDecoder: MappedEncoding[Resource.Id, UUID] = MappedEncoding[Resource.Id, UUID](_.value)
given resourceKindEncoder: MappedEncoding[ResourceKind, String] =
MappedEncoding[ResourceKind, String](_.toString.toLowerCase)
given resourceKindDecoder: MappedEncoding[String, ResourceKind] =
MappedEncoding[String, ResourceKind](s => ResourceKind.valueOf(s.capitalize))
end ResourceSchemaThere's another way to
io.getquill.MappedEncoding is our mate here. We have to define a two-directional encoding (that'll also be used as decoder) for each of the types that fall outside the common types for which quill have already implementations. For our Resource.Id newtype we used a simple apply/value map and contramapping. For our ResourceKind, I have followed a rather naive implementation that simply lower-case or capitalize the string we get from the database to make it a proper ResourceKind
The rest of schemas (i.e. Author's and Category's) follow the exact same principles, so I won't bother you with repeated code, but you can check it out at the post's code repository.
Sticking to the layer pattern to build applications. We create interfaces and using them instead of the actual implementation, implementation of which has some dependencies in their primary constructor.
// Interface
trait ResourcesRepo:
def listResources: Stream[RepositoryError, Resource]
// Implementation
case class ResourcesRepoLive(protected val quill: Quill.Sqlite[SnakeCase])
extends ResourcesRepo
with ResourceSchema:
def listResources: Stream[RepositoryError, Resource] = ???See how much more interconnections has the implemtation, for example, the schemas and the quill instance. This interconnections translate to dependencies to our code. We delay the moment to think about how we're gonna build those dependencies – or even think which ones are we going to need, for that matter.
In other languages, this introduces a new whole world of complexity and uncertainty: Dependency Injection. Here on the other hand you have
Our Quill.Sqlite[?] will be the cornerstone of this layer. With this instance and everything that exports io.getquill we can create query descriptions that will be translated, at compile time, into SQL queries.
Let's explore our main repository: ResourcesRepo.
In ZIO, is customary to implement these as a class who extends this interface plus the suffix live. We'd need to include our schema too, so when we're building queries quill can know which decoder/encoder use for a type it doesn't know, like our newtypes we introduce earlier in our models.
class ResourcesRepoLive(protected val quill: Quill.Sqlite[SnakeCase])
extends ResourcesRepo
with ResourceSchema
with CategorySchema
with AuthorSchema:
import quill.*To access a table in our queries, we need a querySchema that maps to a table name.
class ResourcesRepoLive(protected val quill: Quill.Sqlite[SnakeCase]):
// ...
inline def resources = quote:
querySchema[ResourceRow]("resources")With this, we can start writing our first, which is going to be def listResources: Stream[RepositoryError, ResourceRow]. Here we have our first interesting feature of Quill, and something I find nice resulting from having zio.stream as a first-class citizen within ZIO.
// ...
override def listResources: Stream[RepositoryError, ResourceRow] =
val expr = quote(resources) // (1)
stream(expr).refineOrDie: // (2)
case e: SQLException => RepositoryError(e)The
refineOrDiecombinator receives a PartialFunction in which we, well, refine an error into another type, in this case an
To get a stream from a query is trivial. First we build the expression (1). Then we run it (2). As we are listing every single resource, we don't need filters. Having a stream instead of a list gives us the flexibility to express differents list processing without having all load into memory all at once. We can build pagination from it if necessary, for example, without having to change our query.
For adding a resource, will need to execute an action rather query something. This is accomplished by calling an action method on our query schema, in this case, insertValue.
override def addResource(data: ResourceData): IO[RepositoryError, ResourceRow] =
val effect =
for
resource <- ResourceRow.fromData(data) // (1)
_ <- run(quote(resources.insertValue(lift(resource)))) // (2)
yield resource
effect
.retryN(2) // (3) Retry is here because the id might be already picked.
.mapError(RepositoryError(_))
end addResourceResourceRow.fromData(data) is a helper function that returns an effect, i.e. a ZIO value, so it needs to be sequenced with the runing of the quoted expression. This is becuase creating a random id is a side-effect, meaning something that is not referencially transparent. Notice how at (2) we need to lift the resource value. This is a requirement from Quill's quotations.
The other interesting fact about this method is that we're going to try (3) three times the effect of creating a ResourceRow (and its id), and running the insert action. I did this to illustrate how easy it is for an effect-based program to retry the same action in case something went wrong, in this case, the possiblity that our random id might already have been picked, a small possiblity, but something we won't want our users to worry about.
Our use-cases include 'removing' a resource. This one is interesting because although we don't wan't to remove the authors associated with it (because they might be associated with another resource), we want to remove the associations to the resource.
Many to many associations require their own table, and therefore their own case class. In this application, they have this signature.
final case class ResourceAuthor(resourceId: Resource.Id, authorId: Author.Id)This means we also need another querySchema for our new table:
inline def resourceAuthors = quote(querySchema[ResourceAuthor]("resource_authors"))
override def removeResource(resourceId: Resource.Id): IO[DomainError, Unit] =
val expr = quote: // (1)
resources.filter(_.id == lift(resourceId)).delete
val deleteAuthorsAssocs = quote: // (2)
resourceAuthors.filter(_.resourceId == lift(resourceId)).delete
transaction(run(expr) <&> run(deleteAuthorsAssocs)) // (3)
.mapError:
case e: SQLException => RepositoryError(e)
.unit
end removeResourceThis showcases perfectly a situation where we want to execute two queries ((1) & (2)), but only commit them if they both are successful at the same time. This is done by using Quill's transaction, which receives an effect that in this case is the zipping of the two deletes, concurrently. Notice how this time the action for each expression is delete.
Now, for retrieving all the authors associated with a resource, we would need a Join. Quill makes this farily easy by using the join operator. There multiple kind of joins, but in this ocassion we need a simple Left Join.
override def getResourceAuthors(resourceId: Resource.Id): Stream[RepositoryError, Author] =
val expr = quote:
for
ra <- resourceAuthors.filter(ra => ra.resourceId == lift(resourceId))
author <- authors.join(a => ra.authorId == a.id)
yield author
stream(expr).refineOrDie:
case e: SQLException => RepositoryError(e)
end getResourceAuthorsFirst we filter those rows at resourceAuthors that has our specified id. Then we join those rows with their respective authors by using query#join.
With this we have completed our ResourceRepo implementation. As I mentioned the rest are simplier than this one, so I'd be a good exercise to implement those by yourself with the tools you learned.
The material store is where all the files associated with resources will end up. The goal is to let the user do something like:
btek material add 8u4xhdIf --name 'My PDF' download.pdf
btek material add 8u4xhdIf video.mp4
btek material list
# Programming
# [8u4xhdIf] Functional Programming in Scala
# - [Yif39s3f] My PDF.pdf
# - [123dfsl9] video.mp4To do this, we want a repository that deals with files and a directory, instead of one that deals
with a database. That'll be the job of our MaterialStore.
First, as a dependency of our implementation, we'd need a path that points to the place the user wants to store their files. If you remember in our config section, we set our configuration to have a directory to save our database. Now we want to use the same dir with a 'material' directory appended.
import zio.nio.file.Path
class MaterialStoreLive(storeDir: Path) extends MaterialStore
object MaterialStoreLive:
val layer = ZLayer:
for
cfg <- ZIO.service[AppConfig]
materialDir = cfg.configDir / "material"
_ <- ZIO.attemptBlocking(materialDir.toFile.mkdir())
yield MaterialStoreLive(materialDir)Now, let's write a method to store a file into our directory.
ZIO nio offers nice interfaces to use Input/Output operations using, well, ZIO.
libraryDependencies += "dev.zio" %% "zio-nio" % zioVersionFrom it we're going to use zio.nio.file.Files, which gives us a lot of tool to work with files (duh).
override def store(
materialId: Material.Id,
name: Material.Name,
file: Path,
): IO[IOError, MaterialSource] =
val destFilename = s"${materialId.value}.${name.value}" // (1)
val destPath = storeDir / destFilename // (2)
val materialSource = MaterialSource(materialId, destPath)
Files
.copy(file, destPath, StandardCopyOption.REPLACE_EXISTING)
.as(materialSource)
.mapError:
case e: IOException => IOError(e)
end storeThe destination will be our storeDir / destinationFilename (2), where destFilename is simply the unique identifier of a material, followed by a dot, followed by the material's name (1). Then, by using Files, we're going to copy the path the user passed to us as input to the destPath (2).
We need a method for walking the materials' directory and read each of the files in there. zio-nio's Files offer a function named walk for exactly that. It receives a path as its parameter and returns a stream of paths. The first of these paths is the directory path we're walking, so we're going to drop it.
override def listMaterialsFiles: Stream[DomainError, MaterialSource] =
Files
.walk(storeDir)
.drop(1)
.map: filePath =>
Material
.Id(filePath.filename.toString.split('.').head)
.map: id =>
MaterialSource(id, filePath) // (1)
.collect:
case Right(material) => material // (2)
.mapError:
case e: IOException => IOError(e)Remember that we're storing each material with a unique identifier. When traversing the material directory, we have to parse back this id into a MaterialId (1). Since the result of this is an Either that is guaranteed to work for the files we saved ourselves, but not other kind of files (e.g. those annoying .DS_Store files macos autogenerate), we have to collect those right values only (2).
For deleting a material, we first need a helper method for getting one from an id. This method will use our list function to get the stream of files, and then we'd just have to filter and get the head.
private def getMaterialFile(id: Material.Id): IO[DomainError, MaterialSource] =
listMaterialsFiles
.find(_.id == id)
.runHead
.someOrFail(MaterialSourceNotFoundError(id))Delete a material consists only in getting the material source from an id (with our helper method), pattern match the source (which is a zio.nio.file.Path), and then use Files.delete with it.
private def deleteMaterial: MaterialSource => IO[DomainError, Unit] =
case MaterialSource(_, source) =>
Files.delete(source).mapError { case e: IOException =>
IOError(e)
}
override def delete(id: Material.Id): IO[DomainError, Unit] =
getMaterialFile(id).flatMap(deleteMaterial)This last layer is describe by Gabriel Volpe in his Practical Functional Programming with Scala as a program. A program uses the infrastructure of our application to model actual use-cases. These use-cases will map 1:1 the commands we established the user must be able to use.
The librarian itself will be just a simple class that receive all our infrastructure on its primary constructor:
class LibrarianLive(
materialStore: MaterialStore,
materialsRepo: MaterialsRepo,
resourcesRepo: ResourcesRepo,
authorsRepo: AuthorsRepo,
categoriesRepo: CategoriesRepo,
):
// ...For a user to create a resource, we have to create the authors and the category associated with it if it doesn't exists yet. If a category or an author is not found, we're making the effect fail with a NotFoundError, so if we catch one of them while fetching the category or author, then we'd need to create a new one from there.
private def createCategoryIfDoesNotExist(name: Category.Name): IO[DomainError, Category] =
categoriesRepo
.getByName(name)
.catchSome:
case NotFoundError(_) =>
categoriesRepo.insert(Category(name)).as(Category(name))
private def createAuthorIfDoesNotExist(name: String): IO[DomainError, Author] =
authorsRepo
.getByName(name)
.catchSome:
case NotFoundError(_) =>
for
id <- Author.Id.make
author = Author(id, name)
_ <- authorsRepo.insert(author)
yield authorWith these helpers, let's add our function for creating resources. We receive authors as a set and a category as an optional.
/** Creates a resource and its associated authors and categeory if they don't not exists, fetching
* them by name
*/
def createResource(
title: String,
kind: ResourceKind,
category: Option[Category.Name],
publicationYear: Option[Short],
authors: Set[String],
): IO[DomainError, ResourceRow] =
val resourceData = ResourceData(title, kind, publicationYear, category)
for
authors <- ZIO.foreach(authors)(createAuthorIfDoesNotExist(_)) // (1)
_ <- ZIO.fromOption(category).foldZIO(_ => ZIO.none, createCategoryIfDoesNotExist(_).asSome) // (2)
resource <- resourcesRepo.addResource(resourceData) // (3)
_ <- resourcesRepo.addAuthorsToResource(resource.id, authors.map(_.id).toSeq*) // (4)
yield resource
end createResource- At (1), we use the ZIO constructor
ZIO.foreach, which takes up a list and applies a ZIO to each one, returning a sequence wrapped in a ZIO effect. - Second (2) converts an option into a zio, so it can be used for applying ZIO effects. We fold it so we can provide a default value in case a category name wasn't passed by the user. If we have a name to work with, we pass it over our helper function for getting or create a category.
- We add the data using our
resourceRepoto create a new resource. - Now we must add the relations we need to create for each author with our resource. The could be done atomically when adding a resource, but that would have make the add method way harder to understand. Using it like this makes it clear that we're using a different datasource, resource_authors.
For a resource to be displayed with all its relevant information, we have to fetch not only the row as it is from the database, but also retreive what's associated with it, namely a category and their authors.
private def resourceFromRow(row: ResourceRow): IO[DomainError, Resource] =
for
category <-
ZIO.fromOption(row.categoryName).foldZIO(_ => ZIO.none, categoriesRepo.getByName(_).asSome) // (1)
authors <- resourcesRepo.getResourceAuthors(row.id).runCollect.map(_.toSet) // (2)
yield Resource(row.id, row.title, row.kind, category, row.publicationYear, authors)Here we use the same pattern as before, where we build a ZIO effect from an option and fold over it (1), providing a none as default and a some if we get a category name to work with. Authors related to a resource are fetched as a stream (2), so we need to runCollect all of them. This gives us a chunk so we have to turn it into a set first (2).
private def listResourcesFromRows(categoryOpt: Option[Category.Name] = None) =
resourcesRepo.listResources(categoryOpt).mapZIO(resourceFromRow(_))With this helper that gives us a stream of Resources, last thing we have to do is to group resources by their category, so they can be displayed contiguously. For this reason and because we need to do the same thing when displaying materials, I created a helper method for streams that uses groupByKey, and just return them aside the key we're gruping with:
import zio.stream.*
import zio.*
extension [R, E, O](s: ZStream[R, E, O])
def simpleGroupByKey[K](f: O => K): ZStream[R, E, (K, O)] = s.groupByKey(f): (k, ss) =>
ss.map(k -> _)Having a stream of grouped tuples, we can create a map from their key to a stream of those grouped records. We'll have another extension method for that:
extension [R, E, K, O](s: ZStream[R, E, (K, O)])
def runCollectMap: ZIO[R, E, Map[K, UStream[O]]] =
s.runFold(Map.empty[K, UStream[O]]):
case (s, (k, t)) =>
s + (k -> (s.get(k).getOrElse(ZStream.empty) ++ ZStream.succeed(t)))With this implementation, we'd have to load all the records in memory, which is not ideal, but gives us the ability to work with them still using streams, which is good for when we build a string of outputs for each resource record.
Now let's pull everything together:
/** @return Resources grouped by category name, `"Root"` being the default. */
def listResources(
categoryOpt: Option[Category.Name],
): IO[DomainError, Map[Category.Name, UStream[Resource]]] =
listResourcesFromRows(categoryOpt)
.simpleGroupByKey(_.category.getOrElse(Category.root).name)
.runCollectMapFor resources without a category we're setting Category.root as fallback.
Deleting a resource is quite simple. We're just going to use our repository method for that:
def deleteResource(resourceId: Resource.Id): IO[DomainError, Unit] =
resourcesRepo.removeResource(resourceId)To add material, we first need to create the material in the database because we need its id to actually store the file in our directory.
def addMaterial(
name: Option[String],
resourceId: Resource.Id,
file: JPath,
): IO[DomainError, Material] =
val filePath = Path.fromJava(file)
for
material <- materialsRepo.addMaterial(MaterialData(name, resourceId, filePath)) // (1)
_ <- materialStore
.store(material.id, material.name, filePath) // (2)
.tapError: _ =>
materialsRepo.deleteMaterial(material.id) // (3)
yield material
end addMaterialWe create the material using the MaterialData case class to group the data (1). Then, using the newly created id, we can store the file we're receiving as a function argument (2). This time, however, we want to make sure no material exists only in the database when, for example, storing fails (which can easily be the case). For this reason, we want to delete the Database material if something goes wrong with the storing function (3). ZIO's tapError makes this quite easy.
All we needed for this one we've already done it. We need to group records by key, the category, and then collect them grouped as a map. The only thing that changes is that for each resource, we need to fetch their materials from the database (1).
def listMaterials(
resourceIdOpt: Option[Resource.Id],
): IO[DomainError, Map[Category.Name, UStream[(Resource, List[Material])]]] =
listResourcesFromRows()
.mapZIO(r => materialsRepo.listMaterials(Some(r.id)) /* (1) */.runCollect.map(r -> _.toList))
.simpleGroupByKey(_._1.category.getOrElse(Category.root).name)
.runCollectMap
end listMaterialsIf you remember, we want to add certain flexibility at the moment of removing materials. We want the user to be able to type something like:
btek material list
# Programming
# [8u4xhdIf] Functional Programming in Scala
# - [Yif39s3f] My PDF.pdf
# - [123dfsl9] video.mp4
# Deletes only the specified material.
btek material remove Yif39s3f
# Removes the specified material under
btek material remove 8u4xhdIf video.mp4
# Removes all material associated with the resource.
btek material remove 8u4xhdIfThese use-cases translates to:
- The user can use a
Material.Idto remove a specific material. - The user can also use a
Resource.Idand a material name to remove a material, by name, associated with the given resource. - The user can remove all materials under a resource by just passing a
Resource.Id, without a name.
Let's say we have the material we're going to remove:
def deleteFromStoreWithRevert(material: Material) = materialStore
.delete(material.id)
.tapError: _ =>
materialsRepo.addMaterial(material) // (1)If the deletion from the store fails, we'd like to restore the material into the database (1). With this helper, if the user passes a Material.Id, then have to try to:
val fromMaterialId =
for
materialId <-
ZIO.fromEither(Material.Id(resourceOrMaterialId)).mapError(e => NewtypeBuildError(e)) // (1)
material <- materialsRepo.getMaterialById(materialId) // (2)
_ <- materialsRepo.deleteMaterial(materialId) // (3)
_ <- deleteFromStoreWithRevert(material) // (4)
yield List(materialId)- Build the id. This might fail if it is not eight characters long.
- Get the material from the database, to ensure it exists at the database.
- Delete it from the database.
- Delete from the store, using our helper that contains the cleaning up in case of an error.
In case the user passes a Resource.Id, things get more complicated.
val fromResourceId =
for
resourceId <-
ZIO.fromEither(Resource.Id(resourceOrMaterialId)).mapError(e => NewtypeBuildError(e)) // (1)
materials <-
materialsRepo
.listMaterials(Some(resourceId)) // (2)
.filter(m => materialNameOpt.fold(true)(materialName => m.name == materialName)) // (3)
.runCollect
.map(_.toList)
.flatMap: // (4)
case Nil =>
ZIO.fail:
NotFoundError(
s"$resourceOrMaterialId${materialNameOpt.fold("")(name => s" $name")}",
)
case list => ZIO.succeed(list)
_ <- ZIO.foreachDiscard(materials)(material => materialsRepo.deleteMaterial(material.id)) // (5)
_ <- ZIO.foreachDiscard(materials)(material => deleteFromStoreWithRevert(material)) // (6)
yield materials.map(_.id)- We build the
Resource.Id. This might fail. - We list all materials associated with a resource.
- We filter by the name we might receive as argument. If none was passed, that means we don't want to filter at all, so a
truewill do. Otherwise, we compare them. - If we get an empty list, that means we didn't find materials to delete and this must be reported to the user. Otherwise we just return the list.
foreachDiscardexecutes a ZIO for each value, but discard their results. We just want to know if it succeds or it fails.- Do the same but with our helper method that reverts each material to the database if the store deletion fails.
Materials repo offers to pass an optional resource Id as a filter to get materials. Is defined as follows:
override def listMaterials( resourceIdOpt: Option[Resource.Id] = None, ): Stream[DomainError, Material] = val expr = resourceIdOpt.fold(quote(materials)): ri => quote: materials.filter(_.resourceId == lift(ri)) stream(expr).refineOrDie(RepositoryError(_)) end listMaterials
I hope this was bearable. With all the interfaces we had to work with we abstracted away the implementation details of each, and focused in compose those into a cohesive program, and that allowed us to keep us busy thinking about the user experience and not much about details of datastores and more. Notice how we choose where and how to deal with error thrown in previous layer, of if we take care of them at all in this layer. This is one of the greatest benefits of using effect systems. Our whole program is wrapped with its context and we choose how much of that context is important for us at any given time.
With all this, we're almost there. The only piece missing so far is the presentation layer, where our users will trigger our program and pass arguments to it.
For our CLI interface, we're going to use the official ZIO library zio-cli. Although I'm not the biggest fan of resorting to macros to do all the dirty work, I have to say that using this library was breeze to use.
To create the structure of nested subcommands we'll use Algebraic Data Types. Let's go through each command and subcommand bit by bit.
We're going to have two main sub-commands, namely resources and materials. For this reason, it makes sense to have a Subcommand ADT with two subclasses.
sealed trait Subcommand extends Product with Serializable
object Subcommand:
sealed trait Resources extends Subcommand
sealed trait Materials extends SubcommandBoth resources and materials have subcommands, so let's create an ADT for each one.
// ...
object Subcommand:
sealed trait Resources extends Subcommand
object Resources:
sealed trait ResourcesSubcommand extends Subcommand
sealed trait Materials extends Subcommand
object Materials:
sealed trait MaterialsSubcommand extends SubcommandThe first command in our pile is create. See how it works:
# create a resource
btek resources create --kind book --title "Functional Programming in Scala" --authors "Michael Pilquist, Runar Bjarnason, Paul Chiusano" --category Programming --published 2024First let's get our ADT for the subcommand.
// ...
object Subcommand:
// ...
object Resources:
// ...
object ResourcesSubcommand:
final case class Create(title: String, kind: ResourceKind, authors: Set[String] = Set.empty)
extends ResourcesSubcommandzio-cli comes with its ZIOAppDefault, ZIOCliDefault. This App wrapper will give us a function to implement: cliApp. This function receives a command builder, and after it's build it'll give us the resulting subcommand case class, which we can pattern match to feed their values into our actual program.
override def cliApp = CliApp
.make("btek", "0.1.0-SNAPSHOT", HelpDoc.Span.text("sample docs"), btek):
case Subcommand.Resources.ResourcesSubcommand.List => ???But how do we describe how we want to build those case classes?
For this purpose, ZIO CLI has two kinds of descriptors: Args & Options. They describe, as it is evident, arguments and options, respectively, a command needs. They both have static constructors that describe a type to be capture by them, such as text, localDate or even directory. They both might be combined with others in kind of the same way: with the ++ operator.
Behind the scenes, this opertor gets translated into a tuple of arguments and options that can be mapped into our case classes.
import zio.cli.*
val resourcesCreate = Command(
"create",
Options.text("title") ++
Options.text("kind").mapTry(s => ResourceKind.valueOf(s.toLowerCase.capitalize))
++ Options.text("category").map(Category.Name(_)).optional
++ Options.integer("publication-year").map(_.toShort).optional
++ Options.text("authors").??("author list separated by commas").alias("author").optional,
).map:
case (title, kind, category, publicationYear, authorsOpt) =>
Subcommand.Resources.ResourcesSubcommand.Create(
title,
kind,
category,
publicationYear,
authorsOpt.map(authors => authors.split(",").map(_.trim()).toSet).getOrElse(Set()),
)Now, when we create the CliApp, we can pattern match the result of the arguments the user passed, converted into our case classes.
// ...
case ResourcesSubcommand.Create(t, k, c, p, a) =>
librarian
.createResource(t, k, c, p, a)
.fold(
e => s"An error has occurred while creating the resource: ${e.message}",
r => s"Resource with id ${r.id} was created",
)
.flatMap(zio.Console.printLine(_))From now on we're going to move fast, since we've already got the basis for building CLI commands and use them. To print a nice list of resources, we gotta create a helper function that receives the Map[Category.Name, UStream[Resource]] we build in the librarian chapter, using our stream helper functions.
Each resource is going to be shown with its authors and year of publication, nicely display and handling the cases where authors or a publication year is not available:
private def showResource(r: Resource): String =
def showMeta: Resource => String =
r =>
val authors = r.authors.map(_.name).mkString(", ")
val publicationYear = r.publicationYear.map(_.toString).getOrElse("")
if !authors.isBlank() || !publicationYear.isBlank() then
s"(${List(authors, publicationYear).filterNot(_.isBlank()).mkString(", ")})"
else ""
val meta = showMeta(r)
List(s"[${r.id}]", r.title, meta).filterNot(_.isBlank()).mkString(" ") ++ "."
end showResourceEach category is displayed as a header, followed by the resources associated with it, with a bit of indentation.
private def showCategory(
categoryName: Category.Name,
resources: UStream[(Resource, List[Material])],
): UStream[String] =
ZStream(categoryName.value.capitalize) ++ resources.flatMap: (r, ms) =>
ZStream("\t- " ++ showResource(r)) ++ ZStream
.fromIterable(ms)
.map(m => "\t\t* " ++ showMaterial(m))For now, let's ignore the fact of displaying material associated with a resource. When we're displaying only resources, that's the same as displaying them with an empty material list associated.
def showResources(resourcesMap: Map[Category.Name, UStream[Resource]]): UStream[String] =
showResourcesWithMaterials(resourcesMap.map((k, v) => (k, v.map(_ -> List.empty))))This is how we show resources with materials:
def showResourcesWithMaterials(
resourcesMap: Map[Category.Name, UStream[(Resource, List[Material])]],
): UStream[String] =
val rootOpt = resourcesMap.get(Category.root.name)
val noRoot = resourcesMap - Category.root.name
val rootShow =
rootOpt.fold(ZStream.empty)(rootResources => showCategory(Category.root.name, rootResources))
rootShow ++ ZStream.fromIterable(noRoot).flatMap(showCategory)
end showResourcesWithMaterialsThe command building looks like this. It receives at most 1 category name (that means it can also receive 0 arguments), so when we build the Option[Category.Name], we'd have to make sure to take only the head as an option.
val resourcesList =
Command("list", Args.text("category").map(Category.Name(_)).atMost(1))
.map(category => Subcommand.Resources.ResourcesSubcommand.List(category.headOption))
.withHelp("Lists all resources or under a single ")We already have what we need to process this command, our Librarian list mehtod.
case ResourcesSubcommand.List(category) =>
librarian
.listResources(category)
.map(CliDisplayer.showResources(_))
.flatMap(s => s.runForeach(zio.Console.printLine(_)))Here we use the showResources method we created previously. Remember that it yields an stream of strings, each corresponding to a line, which we print to the console.
To remove resources, we expose the remove command who receives a resource id, which we can map right inside the command so zio-cli can report the error to the user if the id is type in a wrong format. We map the left value from a BuildFailure which is the type of monix.newtype failures, to a HelpDoc, which is a sort of DSL for building documentation that can be displayed with some format by ZIO CLI.
val resourcesRemove =
Command(
"remove",
Args
.text("resource-id")
.mapOrFail(Resource.Id(_).left.map(e => HelpDoc.p(e.toReadableString))),
).map(id => Subcommand.Resources.ResourcesSubcommand.Remove(id))Processing this command require us to just call the delete method from our Librarian.
// ...
case ResourcesSubcommand.Remove(resourceId) =>
librarian
.deleteResource(resourceId)
.map(_ => s"Resource $resourceId was successfully deleted.")
.flatMap(zio.Console.printLine(_))With that, we're completely set to manage resources from our CLI! There's one last piece missing, Materials.
Materials are the main reason for this app to exists as a CLI. It makes easy to manage files associated with a resource we've been watching or reading.
Add has an interesting argument from the CLI point of view. It receives a file path as its last argument. This can be given in absolute or relative terms just as with any other CLI program we use. We can instruct ZIO CLI to make sure this file actually exists or fail otherwise by passing a zio.cli.Exists argument to the Args.file descriptor. Exists is simply a boolean which purpose, I suppose, is to make very very clear your intentions.
val materialsAdd =
Command(
"add",
Options.text("name").optional,
Args
.text("resource-id")
.mapOrFail(Resource.Id(_).left.map(e => HelpDoc.p(e.toReadableString)))
++ Args.file("file", Exists.Yes),
).map:
case (name, (resourceId, file)) =>
Subcommand.Materials.MaterialsSubcommand.Add(name, resourceId, file)Appart from that, this has nothing you haven't already saw in this tutorial. The actionable part is also quite boring (a good thing while programming).
// ...
case MaterialsSubcommand.Add(name, resourceId, file) =>
librarian.addMaterial(name, resourceId, file)We're going to fly over materials list since we already prepared the ground while listing resources. Remember we created a couple of helper functions to group and then collect streams into a map from a key? We'll do that again, but this time, collecting the material associated with each resource. The command itself receives an optional Resource.Id, which serves to filter and show material only for a single resource.
val materialsList = Command(
"list",
Args
.text("resource-id")
.mapOrFail(Resource.Id(_).left.map(e => HelpDoc.p(e.toReadableString)))
.atMost(1),
).map: resourceId =>
Subcommand.Materials.MaterialsSubcommand.List(resourceId.headOption)When receiving a list command, we process it like this:
case MaterialsSubcommand.List(resourceId) =>
librarian
.listMaterials(resourceId)
.flatMap: resourcesMap =>
CliDisplayer
.showResourcesWithMaterials(resourcesMap)
.runForeach(zio.Console.printLine(_))We use the same displayer as before: showResourcesWithMaterials. This 'displayer' has the peculiarity of displaying, surprise, materials.
private def showMaterial(m: Material): String =
s"[${m.id}] ${m.name}"Remove is our last use-case. Yay!
Here, we're not going to parse the id receive from the CLI directly because we don't know if it is either a Resource.Id or a Material.Id. We also receive an optional material name which will only be used in case we get a Resource.Id as first argument (because getting a material id means we are certain which exact material to delete). Again, we model optional arguments by using the atMost(1), because ZIO CLI doesn't support it any other way.
def materialsRemove = Command(
"remove",
Args
.text("resource-or-material-id") ++ Args.text("material-name").map(Material.Name(_)).atMost(1),
).map: (resourceOrMaterialId, materialName) =>
Subcommand.Materials.MaterialsSubcommand.Remove(resourceOrMaterialId, materialName.headOption)Grouping all material subcommands needs to use the same method as with resources, subcommands:
val materials = Command("materials")
.withHelp("help")
.subcommands(materialsAdd, materialsList, materialsRemove)The program when removing goes as usual, with reporting of each material that was deleted:
case MaterialsSubcommand.Remove(resourceOrMaterialId, materialName) =>
librarian
.removeMaterial(resourceOrMaterialId, materialName)
.map: mis =>
"The following materials were deleted:\n" ++ mis
.map(mi => s"\t* Material ${mi.value}\n")
.mkString
.flatMap(zio.Console.print(_))We gotta glue all the commands we created and map into a sealed tree by using the subcommands method we used for materials and resources.
val btek = Command("btek", Options.none, Args.none).subcommands(resources, materials)As shown previously, this make all commands to fit into the tree like subcommand structure, which then we could map into the little programs, each we've already shown.
override def cliApp = CliApp
.make("btek", "0.1.0-SNAPSHOT", HelpDoc.Span.text("sample docs"), btek): cmd =>
val program =
ZIO
.service[LibrarianLive]
.flatMap: librarian =>
cmd match
case // Subcommands =>I added another command: setup. This command will run the setupDatabase function, since is quite resource consuming and we want to avoid running it everytime the user uses the application. The implementation has nothing of interest rather than match the command and run the function, so I won't show it here.
Our val program = ... needs every dependency into place to work. To get the service LibrarianLive we need a librarian layer. This layer needs all of our infrastructure layer , which compresses our repositories and our store. These layers, in turn, need of the app layer, which contains our datasource and configuration layers.
This sort of tree like structure can be expressed simply by writing
program.provide(
AppConfig.layer,
dataSourceLayer,
sqliteLayer,
MaterialStoreLive.layer,
MaterialsRepoLive.layer,
ResourcesRepoLive.layer,
AuthorsRepoLive.layer,
CategoriesRepoLive.layer,
LibrarianLive.live,
)I like an approach that makes clear how each layer depends upon a previous one, resembling the tree structure that actually is. For that, layers support syntax to express this kind of dependencies trees. This is what I came up with for this example:
val appLayer = AppConfig.layer >+> dataSourceLayer >+> sqliteLayer
val infraLayer = appLayer >>>
(MaterialStoreLive.layer ++
MaterialsRepoLive.layer ++
ResourcesRepoLive.layer ++
AuthorsRepoLive.layer ++
CategoriesRepoLive.layer)
val programLayer = infraLayer >>> (LibrarianLive.live ++ appLayer)I hope some of my limited knowledge has been successfully passed to you through this article. My intention was to showcase how some ZIO features makes so easy to approach software development with layers, streams, fallbacks and retries, by coding something I found fun to program. The way I learn best is by detailed explainations, using interesting to keep my attention, of a limited set of features, because once I got in the vibe I can pick up the rest by myself, and that's what tried to do with this article.
We used, rather thoroughly, several key libraries in the ecosystem, namely ZIO's Quill, and in case you see value in creating CLIs, which I do since I've already shipped one for production usage, ZIO CLI.
This program, although complete and, let's say, 'useful', is not without its shortcomings and deficiencies. I highly encorage the reader to see this as an opportunity to extend and improve the code that today was presented. Here are a few ideas:
- Cleanup materials in the database for which their actual files has been removed or moved out, and vice versa. This would need another
cleanupcommand that compares what's inside the actual folder and the database and cleans what doesn't match. Tools like brew have something alike. - Add a command for openning material files. This can be quite easy to add for a single platform. For example, on macos a simple
opencommand will do, but the task will get complicated when we consider multi-platforming. - Testing! I must ask for an apology: I totally neglected tests in this application. I know I know, that's bad engineering and all, but I wanted to priotize explaining the core concepts that mades up a ZIO application.
- Interactivity: At the begining I intended this application to have something more concurrent and interactive by providing a Terminal User Interface (TUI) instead of just a CLI, but that quickly became a bit too big to chew in article with this scope, and my exposicion skills.