noosfero | Migration to article invalid in Suggest Article (!1232)

[Rodrigo Souto]

Rodrigo Souto started a new discussion on db/migrate/20170607131552_update_suggest_article_data.rb:

> +class UpdateSuggestArticleData < ActiveRecord::Migration
> +  def change
> +  say_with_time "Updating suggest article datas..." do
> +    Task.all.each do |task|

`Task.all` loads every task on the database in the memory. When dealing with queries on big data scopes (> 10k results) this is very slow and resource consuming. In this cases, you should try to narrow down as much as you can the number of results by using sql filters (known as `scopes` in rails). This is a very common problem on migrations.

So instead of using `Task.all` and then later filter `task.type == "SuggestArticle"` which will load every task into memory to then choose only the ones within that type, you can do `SugestArticle.all` or `Task.where(type: 'SuggestArticle')` (they are the same). This will load into the memory only the results that match that query. We can narrow the results even more by adding the `pending` scope which will filter tasks that are still pending. Ideally we'd also like to replace the `task.data[:article].nil?` filter by a sql equivalent, but since this is a serialized attribute, is is very hard and messy. The same about setting those new values after. If this attributes were regular table columns we'd try to do all this changes through sql queries (`SELECT` and `UPDATE`).

Another cool feature rails offers to mitigate this memory loading problem is the use of `find_each` instead of `each`. `find_each` will load objects 1000 at a time, instead of loading all the results. It can be very usefull in this case as well.

So it'd be a change more or less like this: `Task.all.each` -> `SuggestArticle.pending.find_each`

Take a look at this reference for further information: http://guides.rubyonrails.org/active_record_querying.html

---
Reply to this email directly or view it on GitLab: https://gitlab.com/noosfero/noosfero/merge_requests/1232#note_32139719
You're receiving this email because of your account on gitlab.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listas.softwarelivre.org/pipermail/noosfero-dev/attachments/20170612/55812031/attachment.html>