Select each month even though the month not exist in mysql table

Question

Let say I have these two tables in mysql.

table1:

date         staff_no
2016-06-10   1
2016-06-09   1
2016-05-09   1
2016-04-09   1

table2:

staff_no    name
1           David

Then, I have this query to get analysis for the staff for each month:

SELECT DATE_FORMAT(table1.date,'%b %Y') as month,COUNT(table1.date) as total_records,table2.name as name
FROM table1 as table1 
LEFT JOIN table2 as table2 on table2.staff_no = table1.staff_no
WHERE table1.staff_no = "1" and date(table1.date) between = "2016-04-01" and "2016-06-30" 
GROUP BY table2.name,DATE_FORMAT(table1.date,'%Y-%m')
ORDER BY DATE_FORMAT(table1.date,'%Y-%m-%d')

This query will output:

month      total_records  name
Apr 2016               1  David
May 2016               1  David
Jun 2016               2  David

But, if I replace the date between "2016-04-01" and "2016-07-31" from the query,it wont show me the July record because it is not exist in table1 which is not what I want. I still want to get result like this:

month      total_records  name
Apr 2016               1  David
May 2016               1  David
Jun 2016               2  David
Jul 2016               0  David   

Anyone expert on this? Kindly help me into this. Thanks!

Answer 1

You can build an inline set of variables representing all the dates you want by using any other table in your system that has AT LEAST the number of months you are trying to represent even though the data does not have to have dates. Just has records that you can put a limit on.

TRY the following statement that uses MySql variables. The FROM clause declares a variable inline to the SQL statement "@Date1". I am starting it with MARCH 1 of 2016. Now, the select fields list takes that variable and keeps adding 1 month at a time to it. Since it is combined with the "AnyTableWithAtLeast12Records" (literally any table in your system with at least X records), it will create a result showing the dates. This is one way of forcing a calendar type of list.

But notice the SECOND column in this select does not change the @Date1 via the := assignment. So, it takes the date as it now stands and adds another month to it for the END Date. If you need a smaller or larger date range, just change the limit of records to create the calendar spread...

select
     @Date1 := date_add( @Date1, interval 1 month ) StartDate,
      date_add( @Date1, interval 1 month ) EndDate
    from
      AnyTableWithAtLeast12Records,
      ( select @Date1 := '2016-03-01' ) sqlvars
   limit 12;

The result is something like...

StartDate   EndDate
2016-04-01  2016-05-01
2016-05-01  2016-06-01
2016-06-01  2016-07-01
2016-07-01  2016-08-01
2016-08-01  2016-09-01
2016-09-01  2016-10-01
2016-10-01  2016-11-01
2016-11-01  2016-12-01
2016-12-01  2017-01-01
2017-01-01  2017-02-01
2017-02-01  2017-03-01
2017-03-01  2017-04-01

Now you have your dynamic "Calendar" completed in one simple query. Now, use that as a basis for all the records you need counts for and format as you had. So take the entire query above as a JOIN to find records within those date ranges... No other queries or stored procedures required. Now, a simple LEFT JOIN will keep all dates, but only show those with staff when WITHIN the between range of per start/end. So ex: greater or equal to 04/01/2016, but LESS THEN 05/01/2016 which includes 04/30/2016 @ 11:59:59pm.

SELECT 
      DATE_FORMAT(MyCalendar.StartDate,'%b %Y') as month,
      COALESCE(COUNT(T1.Staff_no),0) as total_records,
      COALESCE(T2.name,"") as name
   FROM 
      ( select @Date1 := date_add( @Date1, interval 1 month ) StartDate,
               date_add( @Date1, interval 1 month ) EndDate
           from
              AnyTableWithAtLeast12Records,
              ( select @Date1 := '2016-03-01' ) sqlvars
           limit 12 ) MyCalendar
        LEFT JOIN table1 T1
           ON T1.Date >= MyCalendar.StartDate
           AND T1.Date < MyCalendar.EndDate
           AND T1.Staff_No = 1
           LEFT JOIN table2 T2
              ON T1.staff_no = T2.StaffNo
   GROUP BY
      T2.name,
      DATE_FORMAT(MyCalendar.StartDate,'%Y-%m')
   ORDER BY 
      DATE_FORMAT(MyCalendar.StartDate,'%Y-%m-%d')

Answer 2

Consider the following schema with the 3rd table being the year/month Helper Table mentioned. Helper tables are very common and can be re-used throughout your code naturally. I will leave it to you to load it up with substantial date data. Note however the way the end date for each month was put together for those of us that want to do less work, while allowing the db engine to figure out leap years for us.

You could have just one column in that helper table. But that would require the use of function calls for end dates in some of your functions and that means more slowness. We like fast.

Schema

create table workerRecords
(   id int auto_increment primary key,
    the_date date not null,
    staff_no int not null
);
-- truncate workerRecords;
insert workerRecords(the_date,staff_no) values
('2016-06-10',1),
('2016-06-09',1),
('2016-05-09',1),
('2016-04-09',1),
('2016-03-02',2),
('2016-07-02',2);

create table workers
(   staff_no int primary key,
    full_name varchar(100) not null
);
-- truncate workers;
insert workers(staff_no,full_name) values
(1,'David Higgins'),(2,"Sally O'Riordan");

Helper table below

create table ymHelper
(   -- Year Month helper table. Used for left joins to pick up all dates.
    -- PK is programmer's choice.
    dtBegin date primary key,   -- by definition not null
    dtEnd date null
);
-- truncate ymHelper;
insert ymHelper (dtBegin,dtEnd) values
('2015-01-01',null),('2015-02-01',null),('2015-03-01',null),('2015-04-01',null),('2015-05-01',null),('2015-06-01',null),('2015-07-01',null),('2015-08-01',null),('2015-09-01',null),('2015-10-01',null),('2015-11-01',null),('2015-12-01',null),
('2016-01-01',null),('2016-02-01',null),('2016-03-01',null),('2016-04-01',null),('2016-05-01',null),('2016-06-01',null),('2016-07-01',null),('2016-08-01',null),('2016-09-01',null),('2016-10-01',null),('2016-11-01',null),('2016-12-01',null),
('2017-01-01',null),('2017-02-01',null),('2017-03-01',null),('2017-04-01',null),('2017-05-01',null),('2017-06-01',null),('2017-07-01',null),('2017-08-01',null),('2017-09-01',null),('2017-10-01',null),('2017-11-01',null),('2017-12-01',null),
('2018-01-01',null),('2018-02-01',null),('2018-03-01',null),('2018-04-01',null),('2018-05-01',null),('2018-06-01',null),('2018-07-01',null),('2018-08-01',null),('2018-09-01',null),('2018-10-01',null),('2018-11-01',null),('2018-12-01',null),
('2019-01-01',null),('2019-02-01',null),('2019-03-01',null),('2019-04-01',null),('2019-05-01',null),('2019-06-01',null),('2019-07-01',null),('2019-08-01',null),('2019-09-01',null),('2019-10-01',null),('2019-11-01',null),('2019-12-01',null);
-- will leave as an exercise for you to add more years. Good idea to start, 10 in either direction, at least.
update ymHelper set dtEnd=LAST_DAY(dtBegin);    -- data patch. Confirmed leap years.
alter table ymHelper modify dtEnd date not null;    -- there, ugly patch above worked fine. Can forget it ever happened (until you add rows)
-- show create table ymHelper; -- this confirms that dtEnd is not null

So that is a helper table. Set it up once and forget about it for a few years

Note: Don't forget to run the above update stmt

Quick Test for your Query

SELECT DATE_FORMAT(ymH.dtBegin,'%b %Y') as month,
ifnull(COUNT(wr.the_date),0) as total_records,@soloName as full_name 
FROM ymHelper ymH 
left join workerRecords wr 
on wr.the_date between ymH.dtBegin and ymH.dtEnd 
and wr.staff_no = 1 and wr.the_date between '2016-04-01' and '2016-07-31' 
LEFT JOIN workers w on w.staff_no = wr.staff_no 
cross join (select @soloName:=full_name from workers where staff_no=1) xDerived 
WHERE ymH.dtBegin between '2016-04-01' and '2016-07-31' 
GROUP BY ymH.dtBegin 
order by ymH.dtBegin; 

+----------+---------------+---------------+
| month    | total_records | full_name     |
+----------+---------------+---------------+
| Apr 2016 |             1 | David Higgins |
| May 2016 |             1 | David Higgins |
| Jun 2016 |             2 | David Higgins |
| Jul 2016 |             0 | David Higgins |
+----------+---------------+---------------+

It works fine. The first mysql table is the Helper table. A left join to bring in the worker records (allowing for null). Let's pause here. That was afterall the point of your question: missing data. Finally the worker table in a cross join.

The cross join is to initialize a variable (@soloName) that is the worker's name. Whereas the null status of missing dates as you requested is picked up fine via the ifnull() function returning 0, we don't have that luxury for a worker's name. That forces the cross join.

A cross join is a cartesian product. But since it is a single row, we don't suffer from the normal problems one gets with cartesians causing way to many rows in the result set. Anyway, it works.

But here is one problem: it is too hard to maintain and plug in values in 6 places as can be seen. So consider below a stored proc for it.

Stored Proc

drop procedure if exists getOneWorkersRecCount;
DELIMITER $$
create procedure getOneWorkersRecCount
(pStaffNo int, pBeginDt date, pEndDt  date)
BEGIN
    SELECT DATE_FORMAT(ymH.dtBegin,'%b %Y') as month,ifnull(COUNT(wr.the_date),0) as total_records,@soloName as full_name
    FROM ymHelper ymH 
    left join workerRecords wr 
    on wr.the_date between ymH.dtBegin and ymH.dtEnd 
    and wr.staff_no = pStaffNo and wr.the_date between pBeginDt and pEndDt
    LEFT JOIN workers w on w.staff_no = wr.staff_no 
    cross join (select @soloName:=full_name from workers where staff_no=pStaffNo) xDerived
    WHERE ymH.dtBegin between pBeginDt and pEndDt 
    GROUP BY ymH.dtBegin
    order by ymH.dtBegin;
END$$
DELIMITER ;

Test the stored proc a number of times

call getOneWorkersRecCount(1,'2016-04-01','2016-06-09');
call getOneWorkersRecCount(1,'2016-04-01','2016-06-10');
call getOneWorkersRecCount(1,'2016-04-01','2016-07-01');
call getOneWorkersRecCount(2,'2016-02-01','2016-11-01');

Ah, much easier to work with (in PHP, c#, Java, you name it). Choice is yours, stored proc or not.

Bonus Stored Proc

drop procedure if exists getAllWorkersRecCount;
DELIMITER $$
create procedure getAllWorkersRecCount
(pBeginDt date, pEndDt  date)
BEGIN
    SELECT DATE_FORMAT(ymH.dtBegin,'%b %Y') as month,ifnull(COUNT(wr.the_date),0) as total_records,w.staff_no,w.full_name
    FROM ymHelper ymH 
    cross join workers w 
    left join workerRecords wr 
    on wr.the_date between ymH.dtBegin and ymH.dtEnd 
    and wr.staff_no = w.staff_no and wr.the_date between pBeginDt and pEndDt
    -- LEFT JOIN workers w on w.staff_no = wr.staff_no 
    -- cross join (select @soloName:=full_name from workers ) xDerived
    WHERE ymH.dtBegin between pBeginDt and pEndDt 
    GROUP BY ymH.dtBegin,w.staff_no,w.full_name
    order by ymH.dtBegin,w.staff_no;
END$$
DELIMITER ;

Quick test of it

call getAllWorkersRecCount('2016-03-01','2016-08-01');
+----------+---------------+----------+-----------------+
| month    | total_records | staff_no | full_name       |
+----------+---------------+----------+-----------------+
| Mar 2016 |             0 |        1 | David Higgins   |
| Mar 2016 |             1 |        2 | Sally O'Riordan |
| Apr 2016 |             1 |        1 | David Higgins   |
| Apr 2016 |             0 |        2 | Sally O'Riordan |
| May 2016 |             1 |        1 | David Higgins   |
| May 2016 |             0 |        2 | Sally O'Riordan |
| Jun 2016 |             2 |        1 | David Higgins   |
| Jun 2016 |             0 |        2 | Sally O'Riordan |
| Jul 2016 |             0 |        1 | David Higgins   |
| Jul 2016 |             1 |        2 | Sally O'Riordan |
| Aug 2016 |             0 |        1 | David Higgins   |
| Aug 2016 |             0 |        2 | Sally O'Riordan |
+----------+---------------+----------+-----------------+

The Takeaway

Helper Tables have been used for decades. Don't be afraid or embarrassed to use them. In fact, trying to get some specialty work done without them is nearly impossible at times.

Answer 3

I would say you need to have RIGHT JOIN here to include staff from second table