nsqd: refactor storage engine

[andyxning]

Fix #1139
Fix #1137

[andyxning]

/cc @mreiferson @ploxiln This PR is ready for review. PTAL.

[ploxiln]

what's not great about github.com/vmihailenco/msgpack is that it has a bunch of dependencies, including google.golang.org/appengine, and that pulls in github.com/golang/protobuf ...

[andyxning]

what's not great about github.com/vmihailenco/msgpack is that it has a bunch of dependencies, including google.golang.org/appengine, and that pulls in github.com/golang/protobuf ...

@ploxiln I think it deserves adding the these dependencies since msgpack will make it faster and denser than json or plain bytes. This will store more messages on the same disk and write to disk more messages.

Actually, when the functionality grows, the project will get more and more complex such as logic and dependency. Kubernetes is an example.

[andyxning]

/cc @ploxiln @mreiferson

[andyxning]

/cc @ploxiln @mreiferson

[andyxning]

/cc @ploxiln @mreiferson

[andyxning]

/cc @jehiah

[andyxning]

/cc @jehiah @mreiferson @ploxiln

[andyxning]

/cc @jehiah @mreiferson @ploxiln

[andyxning]

/cc @jehiah @mreiferson @ploxiln Anyone can help to take a look at this? It has waited for more than 24 days. :(

[ploxiln]

There are msgpack libraries which do not have dependencies.

While msgpack is more efficient than json, especially for numbers and byte buffers, this is still using strings for keys, instead of a single byte. So, I'm still not a fan of this approach.

[andyxning]

While msgpack is more efficient than json, especially for numbers and byte buffers, this is still using strings for keys, instead of a single byte. So, I'm still not a fan of this approach.

Using single byte for keys will force us to manage parsing keys manually and it is obviously error-prone. This is even make the parsing logic more complicated when a new field added. By using json or msgpack we offload this parsing logic to json or msgpack and they have done quite well in marshal and unmarshal in normal cases and abnormal cases such as some field loss or newly added. We need a tradeoff between maintainability and storage efficiency. In this case, i prefer maintainability.

Btw, @ploxiln thanks for reviewing this. Can you reach @mreiferson @jehiah personally for informing them help to review this PR. This PR has been made for more than 30d. :(

[ploxiln]

@mreiferson recently started at a new job/employer, and is waiting to get official authorization to contribute to NSQ

[andyxning]

OK, let's be more patient. :)

[andyxning]

@mreiferson Ping.

[andyxning]

@ploxiln @mreiferson We should work on this PR for now. :)

[andyxning]

We need to push this PR forward. @ploxiln Do you know who is also actively working on nsq. We can discuss this on slack or mailing list. Anyway, we need to support this ASAP. This has been delayed by more than one month. :(

[andyxning]

@ploxiln @jehiah @mreiferson Can you please take a look at this.

Internally, i have upgraded a cluster and everything works fine.

[ploxiln]

But you're running a fork of nsq 0.3.8 right? So it doesn't really benefit your organization much if this is merged?

(I'm curious why 0.3.8 - which of the interfaces removed in 1.0 are you dependent on?)

[andyxning]

which of the interfaces removed in 1.0 are you dependent on?

Nothing. We just run 0.3.8 well and nothing new feature in versions after 0.3.8 we need.

[andyxning]

But you're running a fork of nsq 0.3.8 right? So it doesn't really benefit your organization much if this is merged?

Yes. But i think with this PR nsqd can support more complicated features easily, such as the compression feature.

[yusank]

Hi, is there any plan to release the new version of nsq with this new storage engine? kind need this feature.

[EvansJahja]

Hi @ploxiln @jehiah @mreiferson
We would benefit from this feature as well and it seems this has been dragging for over 2 years now. Are there any plans on continuing on this?