Flyspray - The bug killer!

  • Status Confirmed   Reopened
  • Percent Complete
    10%
  • Task Type Bug Report
  • Category Backend/Core
  • Assigned To No-one
  • Operating System All
  • Severity High
  • Priority Low
  • Reported Version 1.0 devel (github master)
  • Due in Version 1.1 devel
  • Due Date Undecided
  • Votes 4
  • Private
Attached to Project: Flyspray - The bug killer!
Opened by Sven Teichmann - 12.01.2011
Last edited by peterdd - 17.08.2016

FS#1673 - Only white screen after upgrade to 1.0 - reasons

After I upgraded to version 1.0 (the upgrade was successful), flyspray only shows a white page (and the source in firefox shows, that the page is completely white).


Please help us finding the roots of these bugs!

Please report what you were exactly doing before this happend, report us your steps made, the used php version, used OS version, and such information.

We think most cases of that “white screen” are relying on the third party vendor libraries behavior we use.
When a library detects an error, sometimes they just call die() or exit; of php, but suppress error messages. So the script just stopped not giving any output to browser.

The dev versions from github use composer for installing the required libraries. We will package them on the final release together and make sure most cases of “white screen” are fixed.

The task blocks this from closing
ID Project Summary Priority Severity Assigned To Progress
1849 Flyspray - The bug killer! FS#1849 - Installer Overhaul Low High
50%
ZaZy commented on 16.03.2011 16:16

Same here! :(
Looking at the code i can see that in scripts/authenticate.php@73 there is this:

  Flyspray::show_error(8);

but there is no show_error method in the Flyspray class... :S

Sven Teichmann commented on 16.03.2011 16:27

I have reinstalled flyspray a few days ago and patched it to 1.0.0 and now it works. I execute flyspray with fcgid (I'm not sure, maybe at the first time with 1.0.0 it was mit mod_php, but I'm not sure).

ZaZy commented on 16.03.2011 17:33

I solved this way (please note that before starting this procedure i had a working database from 0.9.9.6):

1. remove any FlySpray files from your server except the *attachment* and *cache* folders and the flyspray.conf.php
2. upload all files from 1.0.0 again
3. run the setup/upgrade.php again

All works now for me.

Andrey Morozov commented on 14.04.2012 13:04

I have the same issue! White screen after upgrading from 9.9.6 to 1.0.0 Solution suggested by ZaZy (doublezed) not working for me!

Franck commented on 28.02.2013 10:09

Same issue here, but the solution suggested by Zazy does not work.

I come from a 0.9.9.7 installation.
Deleted all files except for attachments and conf
Copied everything from last build
Ran /setup/upgrade.php
⇒ blank page, nothing in the logs (at least the ones I have access to)

Franck commented on 02.03.2013 06:55

I still have this exact problem, and the solution given does not work

Project Manager
peterdd commented on 17.02.2015 07:13

There are several die(); without a error message in the code.

And some calls with @ in front to suppress error messages.

For example on a local install (opensuse+apache2+php5) where mysqlnd is available but I forgot to install php-mysqli
i got a white page because in vendor/adodb/adodb-php>nano adodb.inc.php in function _Execute

} else {

                      $this->_queryID = @$this->_query($sql,$inputarr);

}

the error message, that the mysqli extension couldn't be loaded, is suppressed.

Project Manager
Psychokiller1888 commented on 06.03.2015 21:31

Vendors do that? Error suppressing is really to be avoided... Maybe time to change vendors? :)

Project Manager
peterdd commented on 06.03.2015 22:51

Probably not the time for that if there is a possibility to resolve that by talking to the "vendors". :-)

I talked for example with oauth2-client guys for fixing a bug in an older version supported by older php5.3.
One of them provided quickly a solution for doing a 0.3.1 release as 0.3.* is the last supported version for php5.3.
I hope his patch is accepted by maintainer for putting as composer releases.

If they do it - great guys.

If not - lame "vendor". :-P

Project Manager
peterdd commented on 10.03.2015 13:06

Well, they suggested that I have to keep my own branch of oauth2-client and are not willing to support php5.3 in their master.

The problem is, I cannot allow me to be sucked deeper and deeper into more projects. I do Flyspray in freetime
(I even took holiday last month for pushing it forward - well I hope I can introduce it in my company, but for now
"my own hat")

That composer stuff was introduced over a year ago into Flyspray source. By that time oauth2-client supported php5.3.
A few months later (april?) they changed the minimal php version to php5.4. BAM!

I wrote code in perl 15 years ago that still works. Two of my hosting providers where I host projects php5.2.* and php5.3.*
are still the default PHP version. In one I can switch as experienced user to php5.4 and php.5.5, in the other to only an currently unsupported and errorful php5.6 version.

Now project Flyspray depends on the willing and personal attitude of some third party guys. :-(

So, what possible solutions for the release of FS1.0 do we have:

  1. Release it with composer requirements as it is now plus FS bugfixes for 1.0
    • We will get "some whining" (or just leave FS) about can't install it because have no access to a terminal/ssh.
    • We will get "some whining" (or just leave FS) about "white" page errors as some components just die() with suppressed error reporting by "@" character.
  2. Release 2 versions:
    • One version with composer as it is
    • One version with the vendor libraries included. This would allow the easiest installation experience for users - no ssh access, no composer installer. We just use the current versions of the vendor libraries, test that they work well with Flyspray and most currently used** php versions (5.2.*-5.6.*) together, fix the small errors if we find some. As long as the vendors library code has no machine depending codes and is just pure php script code it should be possible.
  3. Drop some vendors or composer complete
    • I fear we must work too often on library issues than working on Flyspray itself, especially in future (years..)

Maybe let's do a survey on mailing list or homepage?

Flyspray's strength was its easy install and use instantly, also for non-technical people.
We should keep going that way.

Project Manager
Psychokiller1888 commented on 10.03.2015 13:15

Cool, you get on my proposition by yourself... We should include the vendors in 1.0, with hotfixes for php 5.3+

That way, FS is ready to work when downloaded... No whining about "missing vendors" error or any other vendor php version incompatibility!

And then later on, for an hypotetical FS v2 maybe we should complety drop vendors and do our own cooking, no dependencies...

As you said, Flyspray moto was: Easy to install and use even for very basic users. Let's keep it that way!

No mailing list survey, that's a dev, meaning Jordan, you, Jouni and me decision to take, not users in my opinion

Jouni Ahto commented on 11.03.2015 18:31

For beta, not time enough remaining to go without composer. For final 1.0 release, I'd add a second download version that has everything necessary included without having to use composer. Beta testers tend to be able to handle a little bit of small things themselves anyway, but not so with users of something called the final version.

Something totally another, Psycho, could you consider changing my group here?

Project Manager
peterdd commented on 11.03.2015 19:27

I added jahto to Developers.

Curious that it worked, because I'm "Developer" only, not "Project manager".

Jouni Ahto commented on 11.03.2015 19:39

Oh that's probably one more bug then that we'll have to fix... If haven't already, it's not the most fresh installation of Flyspray itself that we are using here. Something must be done to that before we release beta.

Project Manager
peterdd commented on 11.03.2015 20:16

Yea, I think with feature freeze or beta someone should immediatly update bugs.flyspray.org to the current dev version as we fixed so many things this year.

Project Manager
Psychokiller1888 commented on 11.03.2015 20:24

Yep, it's planned to update

Project Manager
Psychokiller1888 commented on 11.03.2015 23:06

Btw, perfect that you added jahto, forgot about it. Now the question is how did you add him, so we can fix if needed :)

About composer, that's what I thought, let's go until release without adding the vendors, and then we'll propose a version including the vendors... Or maybe a link to download the vendors pack or something like that...

Project Manager
peterdd commented on 12.03.2015 02:25

Ok.

Jouni Ahto commented on 12.03.2015 20:47

Pre-prepared separate downloadable vendor pack for final release is a good solution too for those without shell access to their hosting provider or otherwise unable/not wanting to use composer. Would also mean that we would have only one release version.

Project Manager
peterdd commented on 24.03.2015 04:20

I think I found another source for white page deads!

While testing with many tasks in a db (~25000) I saw that get_task_list() in includes/class.backend.php doesn't use the limit and offset parameters for the sql query!!!
And only used later in php for doing permission filtering of tasks.

That means all task data is passed to $tasks resulting in consuming much memory on bigger installations.

SELECT t.*,
 lc.category_name AS category_name,  COUNT(DISTINCT vot.vote_id) AS num_votes, 
p.project_title, p.project_is_active, 
lst.status_name, lt.tasktype_name, lr.resolution_name
FROM {tasks} t
LEFT JOIN  {projects} p ON t.project_id = p.project_id
LEFT JOIN  {list_tasktype} lt ON t.task_type = lt.tasktype_id
LEFT JOIN  {list_status} lst ON t.item_status = lst.status_id
LEFT JOIN  {list_resolution} lr ON t.resolution_reason = lr.resolution_id 
LEFT JOIN  {list_category} lc  ON t.product_category = lc.category_id 
LEFT JOIN  {votes} vot ON t.task_id = vot.task_id 
LEFT JOIN  {assigned} ass ON t.task_id = ass.task_id 
LEFT JOIN  {users} u ON ass.user_id = u.user_id 
WHERE t.project_id = ? AND ( is_closed <> '1' )
GROUP BY t.task_id

ORDER BY t.task_id desc, task_severity desc, t.task_id ASC
LIMIT 5000

emalloc:6.625168 MB
   real:7.077888 MB
$tasks = $db->fetchAllArray($sql)
emalloc:62.328584 MB
   real:62.390272 MB

Here my limit seems to be 64 MB (in fact 1024 * 64000 Bytes=65.536 MB) by hosting provider, in my demo case LIMIT 5272 works, LIMT 5273 white page dead.

Possible Solutions:

  • not using fetchAllArray(), but fetch in a loop until the limit of showable tasks for the user is reached (with correct offset for paging!)

or

  • rewrite the query to contain limit and offset. A downside of this: Result could then if the user listsize limit is for example 50, only a few of this 50 can be shown due user-task permissions.

or

  • rewrite the query to contain limit and offset with checking user-task permissions for the tasks in the query. (I would prefer it if possible with permission checking)

I will add some stuff for easier switchon/off debugging in class.backend.php.

ps: needs fixing of useless double sorting on t.task_id

ORDER BY t.task_id desc, task_severity desc, t.task_id ASC
LIMIT 5272
Jouni Ahto commented on 24.03.2015 09:45

Looked into class.database.php. Guess what? I found this:

public function Query($sql, $inputarr = false, $numrows = -1, $offset = -1)

And it then uses adodb's function SelectLimit if $numrows and $offset are given, and that function is overwritten by different database drivers. The question is if $offset and $perpage in call to get_task_list are correct. Will test immediately.

Jouni Ahto commented on 24.03.2015 10:33

By the way it's not used only for permissions checking, but also to show the tasks itself in tasklist. So, after permissions checking, tasklist could contain fewer entries than tasks per page in user preferences. Also found a big problem: we use the total amount of tasks filling search criteria for displaying "Showing tasks 1 - 50 of 108 Page 1 of 3 – 1 - 2 - 3 - Next > " at the bottom of the page. Which makes using $offset and $perpage impossible, then total would always be at maximum $perpage. Unless we do the query twice, first just to get the total count of tasks fitting the query. Not the best idea in the world, although at least much of the data would already be very warm and probably in database internal caches when doing the query second time.

Jouni Ahto commented on 24.03.2015 11:22

Got a solution ready using the worlds not best idea, but at least it works. Won't commit or make a pull request, others should review this first and perhaps come up with better ideas.

Project Manager
peterdd commented on 24.03.2015 13:56

didn't could see patch on mobile now, but the same problem i had with an ecommerce software.

I solved it there with 2 queries, this first just an outer select count(*) around the whole, the second uses limit and offset.

Due only getting count(*) the db can optimize to return only needed things.
And for the second nearly same query the db is 'warm' and can probably profit from some internal caching.

Jouni Ahto commented on 24.03.2015 15:27

Exactly my thoughts and exactly the way I solved the problem. Doing 2 queries is always preferable to crashing due to lack of memory. I've already tested the patch, seems to be working ok, but I still wait ok for someone else too before committing and pushing.

But it's only first part of solving the whole problem to use offset and limit. The next part is putting as much as possible of permissions checking into the query itself. That might be a little bit harder. Should be done in several small steps and with a check every time that it doesn't break something once again.

Project Manager
peterdd commented on 24.03.2015 19:52

Just put your patch in a new branch on github, so we can work on that. So it can be tested by us and if all ok merged to master.

If I remember I did this years ago: 'SELECT COUNT (*) FROM SELECT originalquery' this way first, thinking the db sql algorithms detect by themselfs they don't need to
"order by" inside the query if the result is only a count(*) and no performance loss here.

I now looked back at that software what I did and see it is currently done by

$count_query = xtDBquery($query);
$count = xtc_db_num_rows($count_query,true);

$this->number_of_rows = $count;
$this->number_of_pages = ceil($this->number_of_rows / $this->number_of_rows_per_page);

So in that software the query is called 2 times. One for counting and pagination, and later again for the view of the visible table page.

The good thing compared to your patch is, it is always the same query and impossible to oversee synchronising query if someone fiddles on the original query.
And maybe due to the exact same query (only missing limit and offset) the "warming" / internal dbcaching can be better used for the second query? Well, I don't know much about internal db optimizing/caching of mysql myisam/innodb or postgresql.

On the other side in your extra written sql for the count some joins can be left out. Only the effort of some handcrafting again if something changes.

Project Manager
peterdd commented on 24.03.2015 20:03

And for the second part, checking the view permissions within the sql query. Yea same opinion, must be careful done to not break / slowdown anything.

We should make a list of what can and need be done for checking permissions and then wisely decide if it can be done for FS1.0 or live with $forbidden_tasks rows in displaying
paginated task list (e.g. constant pagination, but only show allowed task on the current page (e.g. 30 of 50 task rows if page size is 50 and 20 task of this 50 subgroup are forbidden for the user)

This reveals the information that there are 20 task hidden to the user (Grrr, why do they hide 20 of 50 tasks from me! Idiots! ;-)) which in a final solution is not revealed to the user.

Jouni Ahto commented on 24.03.2015 20:36

Ok, I also opened a new branch on github. You know where to look and should guess it by name.

I wouldn't even try to leave any joins out of the query when trying to get just a count of the possible result set. Too big a risk of getting it wrong. And I also nowadays quite trust many database engines query optimizers, they do a really good job in optimizing anything out of the query that is not absolutely needed for getting the final result set.

Jouni Ahto commented on 22.04.2015 17:08

My opinion: leave this one open and move due version forward. We can never be sure which one of our third-party libraries might also cause it, just to struggle against it happening.

Project Manager
peterdd commented on 31.07.2015 05:58

One more example is trying to access a non-existent project:

e.g. https://bugs.flyspray.org/?project=100

instead send a 404 http response page with a message?

Project Manager
peterdd commented on 17.08.2016 20:36

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing