Can't upload images to Redmine anymore

Question

For some strange reason I can't upload images to tickets in Redmine anymore. I can upload a txt file or zip files. When I upload an image in the ticket it either says "Service Unavailable" or "Unprocessable". Weird thing is that it used to work. We updated to the latest Redmine (2.6.0.stable)

I looked at the production.log and this is the error (Can't verify CSRF token authenticity):

Started POST "/uploads.js?attachment_id=1&filename=test.png" for xx.xx.xxx.xxx at 2014-12-03 12:58:49 -0500
Processing by AttachmentsController#upload as JS
Parameters: {"attachment_id"=>"1", "filename"=>"test.png"}
WARNING: Can't verify CSRF token authenticity
Filter chain halted as :verify_authenticity_token rendered or redirected
Completed 422 Unprocessable Entity in 2.6ms (ActiveRecord: 0.3ms)

Here is my Redmine Information:

Default administrator account changed	   True
Attachments directory writable	           True
Plugin assets directory writable	   True
RMagick available (optional)               Exclamation
ImageMagick convert available (optional)   True

Environment:
  Redmine version                2.6.0.stable
  Ruby version                   1.9.3-p547 (2014-05-14) [x86_64-linux]
  Rails version                  3.2.19
  Environment                    production
  Database adapter               Mysql2
SCM:
  Git                            1.8.2.1
  Filesystem                     
Redmine plugins:
  redmine_agile                  1.3.2
  redmine_ckeditor               1.0.16
  redmine_github_hook            2.1.0
  redmine_my_page_queries        2.1.6
  redmine_theme_changer          0.1.0

Answer 1

It turns out that this was a Varnish Issue. We got around this problem by adding this Varnish rule:

if (req.http.host ~ "my\.domain\.com$") {
   return (pipe);
}

Here are some debugging things we did to try to figure out the problem.

1. Temporarily added config.action_controller.allow_forgery_protection = false to application.rb. When we tried to upload an image it I get a Popup: login required for Server on Redmine API. This gave me a clue that it must have been some kind of server issue.

2. Created additional_environment.rb and enabled config.log_level = :debug. This added more debug info to the log file.

Started POST "/uploads.js?attachment_id=1&filename=Screen%20Shot%202014-12-11%20at%2010.01.49%20AM.png" for xx.xx.xxx.xxx at 2014-12-11 11:07:41 -0500
Processing by AttachmentsController#upload as JS
  Parameters: {"attachment_id"=>"1", "filename"=>"Screen Shot 2014-12-11 at 10.01.49 AM.png"}
  ^[[1m^[[35m (0.3ms)^[[0m  SELECT MAX(`settings`.`updated_on`) AS max_id FROM `settings`
  ^[[1m^[[36mSetting Load (0.3ms)^[[0m  ^[[1mSELECT `settings`.* FROM `settings` WHERE `settings`.`name` = 'rest_api_enabled' LIMIT 1^[[0m
  ^[[1m^[[35mAnonymousUser Load (0.3ms)^[[0m  SELECT `users`.* FROM `users` WHERE `users`.`type` IN ('AnonymousUser') LIMIT 1
  Current user: anonymous
  ^[[1m^[[36mSetting Load (0.3ms)^[[0m  ^[[1mSELECT `settings`.* FROM `settings` WHERE `settings`.`name` = 'login_required' LIMIT 1^[[0m
  ^[[1m^[[35mSetting Load (0.2ms)^[[0m  SELECT `settings`.* FROM `settings` WHERE `settings`.`name` = 'force_default_language_for_anonymous' LIMIT 1
  ^[[1m^[[36mSQL (1.2ms)^[[0m  ^[[1mSELECT `members`.`id` AS t0_r0, `members`.`user_id` AS t0_r1, `members`.`project_id` AS t0_r2, `members`.`created_on` AS t0_r3, `members`.`mail_notification` AS t0_r4, `projects`.`id` AS t1_r0, `projects`.`name` AS t1_r1, `projects`.`description` AS t1_r2, `projects`.`homepage` AS t1_r3, `projects`.`is_public` AS t1_r4, `projects`.`parent_id` AS t1_r5, `projects`.`created_on` AS t1_r6, `projects`.`updated_on` AS t1_r7, `projects`.`identifier` AS t1_r8, `projects`.`status` AS t1_r9, `projects`.`lft` AS t1_r10, `projects`.`rgt` AS t1_r11, `projects`.`inherit_members` AS t1_r12, `roles`.`id` AS t2_r0, `roles`.`name` AS t2_r1, `roles`.`position` AS t2_r2, `roles`.`assignable` AS t2_r3, `roles`.`builtin` AS t2_r4, `roles`.`permissions` AS t2_r5, `roles`.`issues_visibility` AS t2_r6 FROM `members` LEFT OUTER JOIN `projects` ON `projects`.`id` = `members`.`project_id` LEFT OUTER JOIN `member_roles` ON `member_roles`.`member_id` = `members`.`id` LEFT OUTER JOIN `roles` ON `roles`.`id` = `member_roles`.`role_id` WHERE `members`.`user_id` = 2 AND (projects.status<>9) ORDER BY projects.name^[[0m
  ^[[1m^[[35mRole Load (0.2ms)^[[0m  SELECT `roles`.* FROM `roles` WHERE `roles`.`builtin` = 2 LIMIT 1
Filter chain halted as :authorize_global rendered or redirected
Completed 401 Unauthorized in 54.3ms (ActiveRecord: 2.7ms)

Current user: anonymous in the log kind of helped lead to the fix.