Skip to main content

Biggish Data

Search This Blog

Building Spelling Correction at Etsy

Get link
Facebook
X
Pinterest
Email
Other Apps

I wrote this a while back on how we set up the spelling correction framework for search at Etsy.

Get link
Facebook
X
Pinterest
Email
Other Apps

Comments

Post a Comment

Popular Posts

Hadoop Counters in Scalding

The idea is to access the cascading FlowProcess through a cascading operator. We use an anonymous function for the operator which does nothing except incrementing a specified hadoop counter. def addCounter(pipe : Pipe, group : String, counter : String) = { pipe.each(() -> ('addCounter)) { fields => new BaseOperation[Any](fields) with Function[Any] { def operate(flowProcess : FlowProcess[_], functionCall : FunctionCall[Any]) { try { flowProcess.asInstanceOf[HadoopFlowProcess] .increment(group, counter, 1L) functionCall.getOutputCollector.add(new Tuple(new Array[Object](1) : _*)) } catch { case cce: ClassCastException => // HadoopFlowProcess is not available in local mode } } } }.discard('addCounter) } In the scalding job, the tuples passing through any part of the Flow can be counted using code like this: .then { p => addCounter(p, ...

Limiting Hadoop Part Files using 'shard' and a Common Gotcha

This is accomplished trivially using the shard(n: Int) method on RichPipe which randomly uses n reducers to redistribute the tuples before writing the files to disk using the write method. A major potential gotcha to watch out for is when reusing a pipe which uses the shard and write combination. If that pipe has more operations performed on it, subsequently, in a different part of the job code, the cascading job planner somehow completely ignores the shard command. This had to bite me many times before I finally realized what was happening. So, in summary, when using the shard in a pipe make sure that the resulting pipe is not used elsewhere in a scalding job. Here's an example of the gotcha where the number of part files generated under mydir will not be 100 as one might expect. (In fact, the number will be 2000.): val writeAndReuse = MySource() .read .groupBy('my-field) { _.reducers(2000).size } .shard(100) .write(Tsv("my-dir") writeAndRe...

Powering Real Time Predictions at Fundera

Introduction We have designed and implemented a flexible and scalable machine learning framework that allows a) rapid development of models offline, b) easy and reliable deployment of models into production, and c) supports multiple frameworks and algorithms. The framework provides a microservice that enables the serving of near real-time predictions from multiple models. Models can be updated with custom frequency. The prediction service is scalable and hence is robust to hardware failure and can grow to support more traffic. We also have extensive monitoring and alerting in place to ensure both model and prediction service performance. General Generalized framework that makes it easy to create a new model by creating a JSON config and sub-classing a base class. Subclass needs to define methods that specify how to fetch the data, how to identify positive negative examples, etc. There are additional methods that can be used to generate custom evaluations on both the tra...

Powered by Blogger