-
Notifications
You must be signed in to change notification settings - Fork 32
feat: add decoding as a multi-threaded CLARA engine #1229
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
baltzell
wants to merge
38
commits into
development
Choose a base branch
from
decoder-engine
base: development
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
22d148b
add convenience method
baltzell 11762ce
remove unused stuff
baltzell 4c319c4
add decoding engine
baltzell 09c09e5
add it to the clara yaml
baltzell a460adf
use a pool
baltzell 11874a9
hmm
baltzell 11f17a8
allow decoder instances to share ConstantsManagers
baltzell 3d741af
share ConstantsManagers
baltzell 27d8cc4
kludge test
baltzell adb6567
cleanup
baltzell 3a91417
try this
baltzell 064fc5c
inherit ConstantsManagers
baltzell 38c124a
cleanup
baltzell 54cefae
only check tables if not shared
baltzell 0d8d854
higher ports on macos, cleanup process dpe process
baltzell 41f3216
remove ineffective pid trap
baltzell b800a37
Revert "higher ports on macos, cleanup process dpe process"
baltzell 746c196
Revert "fix job name"
baltzell acc7d41
Revert "decouple ubuntu/macos builds to reduce wait"
baltzell 2c61171
Revert "remove example engine"
baltzell 37fd575
restore reported data type
baltzell 2f4cde4
rename class
baltzell 900b8b3
undo
baltzell 1d32a0d
fix rebase oops
baltzell 3f9994c
rename class
baltzell a1b5e4a
fix rebase oops
baltzell c56a514
stf
baltzell d8c4f2f
just events, not per thread
baltzell 5380c80
reduce constants sharing
baltzell 91d7a92
bugfix
baltzell b791b24
share one RCDBManager
baltzell ed10100
bugfix
baltzell aec8922
restore
baltzell e15903b
DecoderEngine: just pass along HIPO events
baltzell 70da57d
cleanup, avoid class variable
baltzell 968d675
remove debugging leftover
baltzell cb67bba
revert to hard-coded byte order
baltzell 8178612
default to no sharing
baltzell File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
113 changes: 113 additions & 0 deletions
113
common-tools/clas-reco/src/main/java/org/jlab/clas/reco/DecoderEngine.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| package org.jlab.clas.reco; | ||
|
|
||
| import java.util.Set; | ||
| import java.util.HashSet; | ||
| import java.nio.ByteBuffer; | ||
| import java.nio.ByteOrder; | ||
| import java.util.concurrent.ArrayBlockingQueue; | ||
| import java.util.concurrent.BlockingQueue; | ||
| import org.jlab.clara.base.ClaraUtil; | ||
| import org.jlab.clara.engine.Engine; | ||
| import org.jlab.clara.engine.EngineData; | ||
| import org.jlab.clara.engine.EngineDataType; | ||
| import org.jlab.clara.engine.EngineStatus; | ||
| import org.jlab.detector.decode.CLASDecoder; | ||
| import org.jlab.io.evio.EvioDataEvent; | ||
| import org.jlab.io.hipo.HipoDataEvent; | ||
| import org.jlab.jnp.hipo4.data.SchemaFactory; | ||
| import org.json.JSONObject; | ||
|
|
||
| /** | ||
| * | ||
| * @author baltzell | ||
| */ | ||
| public class DecoderEngine implements Engine { | ||
|
|
||
| static final int POOL_SIZE = 64; | ||
| static final Set<EngineDataType> ED_TYPES = ClaraUtil.buildDataTypes( | ||
| Clas12Types.EVIO,Clas12Types.HIPO,EngineDataType.JSON,EngineDataType.STRING); | ||
|
|
||
| SchemaFactory schema; | ||
| BlockingQueue<CLASDecoder> pool; | ||
| int constantsShared = 64; | ||
|
|
||
| public DecoderEngine() { | ||
| schema = new SchemaFactory(); | ||
| schema.initFromDirectory(System.getenv("CLAS12DIR") + "/etc/bankdefs/hipo4"); | ||
| } | ||
|
|
||
| @Override | ||
| public Set<EngineDataType> getInputDataTypes() { return ED_TYPES; } | ||
| @Override | ||
| public Set<EngineDataType> getOutputDataTypes() { return ED_TYPES; } | ||
| @Override | ||
| public EngineData executeGroup(Set<EngineData> set) { return null; } | ||
| @Override | ||
| public Set<String> getStates() { return new HashSet<>(); } | ||
| @Override | ||
| public String getDescription() { return "decoder engine"; } | ||
| @Override | ||
| public String getVersion() { return "1.0"; } | ||
| @Override | ||
| public String getAuthor() { return "baltzell"; } | ||
| @Override | ||
| public void reset() {} | ||
| @Override | ||
| public void destroy() {} | ||
|
|
||
| @Override | ||
| public EngineData configure(EngineData ed) { | ||
| JSONObject json = new JSONObject(ed.getData()); | ||
| pool = new ArrayBlockingQueue<>(POOL_SIZE); | ||
| CLASDecoder d0 = null; | ||
| for (int i=0; i<POOL_SIZE; i++) { | ||
| CLASDecoder d; | ||
| if (i % constantsShared == 0) { | ||
| d0 = new CLASDecoder(); | ||
| if (json.has("variation")) d0.setVariation(json.getString("variation")); | ||
| if (json.has("timestamp")) d0.setVariation(json.getString("timestamp")); | ||
| d = d0; | ||
| } | ||
| else { | ||
| d = new CLASDecoder(d0); | ||
| } | ||
| pool.add(d); | ||
| } | ||
| return ed; | ||
| } | ||
|
|
||
| @Override | ||
| public EngineData execute(EngineData input) { | ||
|
|
||
| EngineData output = input; | ||
|
|
||
| // if it's EVIO, decode it, otherwise just pass it along | ||
| if (input.getMimeType().equals("binary/data-evio")) { | ||
| EvioDataEvent evio; | ||
| try { | ||
| ByteBuffer bb = (ByteBuffer) input.getData(); | ||
| //evio = new EvioDataEvent(bb.array(), bb.order()); | ||
| evio = new EvioDataEvent(bb.array(), ByteOrder.LITTLE_ENDIAN); | ||
| } catch (Exception e) { | ||
| String msg = String.format("Error reading input event%n%n%s", ClaraUtil.reportException(e)); | ||
| output.setStatus(EngineStatus.ERROR); | ||
| output.setDescription(msg); | ||
| return output; | ||
| } | ||
| HipoDataEvent hipo; | ||
| try { | ||
| CLASDecoder d = pool.take(); | ||
| hipo = new HipoDataEvent(d.getDecodedEvent(evio),schema); | ||
| pool.put(d); | ||
| output.setData("binary/data-hipo", hipo.getHipoEvent()); | ||
| } catch (Exception e) { | ||
| String msg = String.format("Error processing input event%n%n%s", ClaraUtil.reportException(e)); | ||
| output.setStatus(EngineStatus.ERROR); | ||
| output.setDescription(msg); | ||
| return output; | ||
| } | ||
| } | ||
|
|
||
| return output; | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.