Symfony2: use flush() or persist() to optimize data handling

Question

Im working with Symfony (const version="2.5.10") and using XAMPP with PHP version of 5.5.19

My problem is I keep on getting out of memory error. Because I think I query thousand of data (rows) in the database. For info, I have a lot of data in the database. I want to use flush() or anything that could be use to optimize my data handling.

Here is my code in my indexAction controller:

public function indexAction(){
  $em = $this->getDoctrine()->getManager();
  $po = $em->getRepository('MatrixEdiBundle:EdiTransactionDetail')->findDocs('850');
  return $this->render('MatrixEdiBundle:Matrix:index.html.twig', array('po' => $po));
}

index.html.twig

{% extends '::layout.html.twig' %}
{# {% include 'MatrixEdiBundle:Matrix:header.html.twig'%} #}
{% block body %}



  
      {% for tran in po %}
        
      {% endfor %}
    
      
        Edi Matrix
        

          Po Numbers
          Trading Partner Id
          PO 855 Acknowledgement
          PO 997 Acknowledgement
          Advance Shipment Notice
          Invoice
          Purchase Order Change
          Order Status
        
      
      
            {{tran.poNumber}}

            {{tran.ediTransaction.senderId}}

            
            {{ render(controller('MatrixEdiBundle:Matrix:matrix', {
            'po_num': tran.poNumber, 'sender_id': tran.ediTransaction.senderId, 'reciever_id': tran.    ediTransaction.receiverId, 'doc_type': 855, 'gs_number': tran.ediTransaction.gsNumber })) }}
            

            
            {{ render(controller('MatrixEdiBundle:Matrix:matrix', {
            'po_num': tran.poNumber, 'sender_id': tran.ediTransaction.senderId, 'reciever_id': tran.    ediTransaction.receiverId, 'doc_type': 997, 'gs_number': tran.ediTransaction.gsNumber })) }}
            

            {{ render(controller('MatrixEdiBundle:Matrix:matrix', {
            'po_num': tran.poNumber, 'sender_id': tran.ediTransaction.senderId, 'reciever_id': tran.ediTransaction.receiverId, 'doc_type': 856, 'gs_number': tran.ediTransaction.gsNumber }))}}
            

            {{ render(controller('MatrixEdiBundle:Matrix:matrix', {'po_num': tran.poNumber, 'sender_id': tran.ediTransaction.senderId, 'reciever_id': tran.ediTransaction.receiverId, 'doc_type': 810, 'gs_number': tran.ediTransaction.gsNumber})) }}
            

            {{ render(controller('MatrixEdiBundle:Matrix:matrix', {'po_num': tran.poNumber, 'sender_id': tran.ediTransaction.senderId, 'reciever_id': tran.ediTransaction.receiverId, 'doc_type':860, 'gs_number': tran.ediTransaction.gsNumber})) }}
            

          {{ render(controller('MatrixEdiBundle:Matrix:matrix', {
        'po_num': tran.poNumber, 'sender_id': tran.ediTransaction.senderId, 'reciever_id': tran.ediTransaction.receiverId, 'doc_type': 870, 'gs_number': tran.ediTransaction.gsNumber
    })) }}

        
    
   



    
        
            
                
                Details
            
      
          Loading, please wait......
          

              
                
                  Loading, please wait…
                
                
              
          

        

      
          
      
    

  


{% endblock %}

{% block javascripts %}
{% javascripts
  'bundles/matrixdoc/js/jQuery.js'
  'bundles/matrixdoc/js/jquery.dataTables.min.js'
  'bundles/matrixdoc/js/dataTables.bootstrap.js'
  'bundles/matrixdoc/js/bootstrap.js'
   %}
  
  
{% endjavascripts %}
{% endblock %}

Nawfal Serrar · Accepted Answer

You should use pagination getting your data with limits and coupled with partial to get your needed fields if you don't need everything.

http://zrashwani.com/pagination-optimization-symfony2-doctrine/#.VV1av_mSwbg

for partial in DQL , simple example for user:

//partial_fields is an array, which can be passed like this $repo->getPartialUser($user_id, array('field1', 'field2', 'etc..');
    public function getPartialUser($id, $partial_fields){
        $qb = $this->_em->createQueryBuilder()
            ->select('partial u.{'.implode(',',$partial_fields).'}')
            ->from('AcmeUserBundle:User', 'u')
            ->where('u.id = :id')
            ->setParameter('id', $id);
        $result = $qb->getQuery()->getOneOrNullResult();
        return $result;
    }

EDIT EXAMPLE CODE :

i will give you an example but you will probably still need to adapt it to your code or maybe it will work out of the box.

First you need a new twig with simple piece of code to show pages numbers for navigation below the table (note that i will not use datatables paginator but we will keep it for table layout rendering) :

pager.html.twig


    
        {% if pagination.page>1 %}
            <<
            <
        {% endif %}
        {#display p numbers only from p-4 to p+4 but don't go <1 or >pages_count#}
        {% for p in range(max(pagination.page-4, 1),
        min(pagination.page+4, pagination.pages_count)) %}
            {{ p }}
        {% endfor %}
        {% if pagination.page>
            >>
        {% endif %}

EdiTransactionDetailRepository:

add a function with your field that you use in your table :

public function  getPagedDocs($page = 1, $limit = 50)
    {
        $qb = $this->_em->createQueryBuilder()
             ->select('partial d.{field1,field2 , etc..},
                       partial et.{field1,field2, etc..}'
              ))//select doc fields and second partial for your join select edittransaction fields if you have other joins add another partial etc.. 
             ->from('YourBundle:DocsEntity', 'd')
             ->innerJoin('d.ediTransaction', 'et')//not sure which join you need i have no clue but i believe you will want an inner one since you will want a doc to have a ediTransaction because i didn't see any checks in your twig for it if null.
             ->setFirstResult(($page - 1) * $limit)
             ->setMaxResults($limit);

        $paginator =  new Paginator($qb, $fetchJoinCollection = false);//for more performance fetchjoincollection to false if you have joined tables
        $paginator->setUseOutputWalkers(false);//for more performance set to false for more information http://www.doctrine-project.org/jira/browse/DDC-2890

        return $paginator;
    }

PS: Change limit to how many records you want to show in each page depending on the load of data you have make it lower if it is slow for 50 this is depend on how much data each row have.

Next the indexAction in your controller :

public function indexAction($page){
  $em = $this->getDoctrine()->getManager();
  $po = $em->getRepository('MatrixEdiBundle:EdiTransactionDetail')->getPagedDocs($page, 50);
  $count = $po->count();
  $pagination = array(
        'page' => $page,
        'route' => 'docs_index_route_name', //i dont know which name you have but it should be the route name of this indexAction
        'route_params' => array()
    );
    if ($max_records > 0)
        $pagination['pages_count'] = max(ceil($count / $max_records), 1);
  return $this->render('MatrixEdiBundle:Matrix:index.html.twig', array('po' => $po,
  'pagination' => $pagination
    ));
}

Routing:

You need to modify the route of your indexAction as we added a parameter page you need to add it for the route too :

index_docs:
    pattern:  /index/{page}
    defaults: { _controller: "YourBundle:Controller:index", page: 1 }

NOTE : you will need to change the names and stuff i dont know you controller names and bundles so normally you will only need to add /{page} to the pattern and , page:1 to your _controller config which is the default value.

Last thing to do is your index.html.twig which need to include our pager.html.twig

include this piece of code after you close your table :

    {% if po|length > 0  and pagination['pages_count'] is defined and pagination['pages_count'] > 0 %}
        {#---------Pager----------#}
        
            {% include 'YourBundle:EntityDoc:pager.html.twig' %}
        
    {% endif %}

NOTE : The include is the path to the file where you create the pager.html.twig i assume you are familiar with how this works.

So if i didn't forget anything this should work out of the box , it should enhance greatly your performance but i am afraid that your render controller there for each row may still be problematic. But try this solution first and see, it depends what you put inside those controllers.

Symfony2: use flush() or persist() to optimize data handling

Answers (1)

EDIT EXAMPLE CODE :

pager.html.twig

EdiTransactionDetailRepository:

Routing:

Related Questions