tag:blogger.com,1999:blog-2481158163384033132.post3246894593681169723..comments2015-10-29T06:24:05.852-07:00Comments on Java Advent Calendar: Big Data the 'reactive' wayCd-MaNhttp://www.blogger.com/profile/05030326541176171725noreply@blogger.comBlogger8125tag:blogger.com,1999:blog-2481158163384033132.post-6677487856030150332014-02-01T09:00:04.452-08:002014-02-01T09:00:04.452-08:00We have different variations of representing a tab...We have different variations of representing a table partition, however we did not make use of a library to implement this. <br />a) simple hashmap <br />b) offheap for large tables (slightly slower query+update) <br />c) redirection to external system (actual data resides in a database or messaging product) for seamles integration of external subsystems<br /><br />We use straight java serialization for queries, this way a query get hotspot compiled and not interpreted. However we also have a JSON based interface for mobile devices/web at the price of ~30% query + max update rate drop.<br /><br />Actually we did not have the need to store historic data, as we bootstrap reference data from a database, since the back end is event sourced, we get intraday recovery from there. Some few tables are saved by doing cyclically snapshots from a dedicated PersisterNode.<br /><br />However if required, it would be easy to add in a node using e.g. Chronicle to dump selected change streams (= transaction history of a table) to ssd in order to be able to recover at LVC-data-level.Rüdiger Möllerhttps://www.blogger.com/profile/03711813786574992852noreply@blogger.comtag:blogger.com,1999:blog-2481158163384033132.post-44156783063074266712014-01-30T14:43:33.666-08:002014-01-30T14:43:33.666-08:00Hi great article a few questions if you have a mom...Hi great article a few questions if you have a moment: <br /><br />What frameworks/libs did you use to hold your LVC<br />What did you use to hold the LVC data and the historic data - did you have temporal based data<br />For the querying api did you have your own custom schema or make use of any published ones eg. json etc.<br /><br />Many thanksUnknownhttps://www.blogger.com/profile/14875616031454083879noreply@blogger.comtag:blogger.com,1999:blog-2481158163384033132.post-91388954519721823872014-01-13T03:13:42.118-08:002014-01-13T03:13:42.118-08:00Esper seems to implement a similar system, but not...Esper seems to implement a similar system, but not that scalable. Note that we separate Query processing (checking filter conditions on a data change) from the raw "store and forward" of the data (LVC) nodes. With esper, the more queries are active concurrently the lower overall system throughput gets. With our systems, throughput is mostly independent from the number of clients.<br />Another thing is data size: If one does not distribute each table amongs all LVC nodes, one runs into limits regarding table size and throughput, as nodes hosting high frequency tables get pressure. With our system write load is distributed evenly.<br />I am not familar with Esper, so correct me if I made wrong assumptions (just checked Website and documentation).Rüdiger Möllerhttps://www.blogger.com/profile/03711813786574992852noreply@blogger.comtag:blogger.com,1999:blog-2481158163384033132.post-42230883646584490902014-01-10T06:26:32.573-08:002014-01-10T06:26:32.573-08:00Have you experimented with Esper? I've used it...Have you experimented with Esper? I've used it for very high rate systems and have been been very impressed. Anonymoushttps://www.blogger.com/profile/06801487091838944476noreply@blogger.comtag:blogger.com,1999:blog-2481158163384033132.post-67260571634784219352014-01-04T05:19:32.321-08:002014-01-04T05:19:32.321-08:00The underlying message bus in this system uses a t...The underlying message bus in this system uses a topic based brokerless reliable UDP messaging. Most JMS implementations do not support reliable multicast broadcasting, but there are some afaik.<br />However as stated in the article, a pure messaging system cannot provide the required features for continuous query, because a pure messaging has no information about current state (e.g. current price of a stock), additionally to support continusous queries, one always needs the previous state and the new state be broadcasted (see "Difference of data access.." point 3).Rüdiger Möllerhttps://www.blogger.com/profile/03711813786574992852noreply@blogger.comtag:blogger.com,1999:blog-2481158163384033132.post-42092988930539698842014-01-03T13:13:01.462-08:002014-01-03T13:13:01.462-08:00Would JMS type message handling with topics/queues...Would JMS type message handling with topics/queues end points be usable in this context? With some filters maybe? Erichttps://www.blogger.com/profile/08819472618318715881noreply@blogger.comtag:blogger.com,1999:blog-2481158163384033132.post-1079594918051066032013-12-31T07:45:42.499-08:002013-12-31T07:45:42.499-08:00We use open source stuff for non-critical parts of...We use open source stuff for non-critical parts of the application. We prefer libraries in favor of frameworks. <br />For the performance critical stuff, it turned out 95% of open source is too slow. Starting the project with JGroups nearly cost me my Job: its not capable to run stable clusters with permanent throughtput (algorthmic issues (NACKACK is bad) and really slow implementation of core functionality). <br />For messaging we now use FastCast in development and functional testing, we use a commercial (excellent!) low latency messaging in production.<br />I am currently building an open source implementation of such a system "real-live", which is currently pre-alpha [temporary hosted in the 'reallive' folder of the fast-cast project :-) ]Rüdiger Möllerhttps://www.blogger.com/profile/03711813786574992852noreply@blogger.comtag:blogger.com,1999:blog-2481158163384033132.post-79293705142020606702013-12-26T08:51:14.021-08:002013-12-26T08:51:14.021-08:00Do you use any off-the-shelf (open source or other...Do you use any off-the-shelf (open source or otherwise) software to implement this architecture or is it all custom code? If so, what? Or is that a secret? :-)Cd-MaNhttps://www.blogger.com/profile/05030326541176171725noreply@blogger.com