I was in about the same place as you about 4 yrs ago.
Ex wanted our son weekdays and I could have him weekends - he was 3yrs old. He tried to reason that I worked and he didn't. Told him no way. and on and on it went.
It sounds like he's lawyered up. You should seek legal advice, as the laws in NZ are most likely different. However some things are likely to be similiar.
You need to set some rules for the kids and (for any legal stuff now or later.)
No 1 - No-one is to slander the other parent in front of the children (and no arguing in front of children). No matter how much your mum / best friend, may not like what your ex is doing - they should not show this in front of the children - no matter what age.
No 2 - keep a diary of conversations, phone calls, events, etc. Including when the kids visit their father (including times, dates), their mood, etc when returning. This is proof of visitation, your ex's contact with children, etc. Do not include your feelings (eg, ex made me feel very mad, i hate him) as if the diary become court evidence - then all entries are evidence, etc.
No 3 - Establish visitaiton and try to keep to it. (if he gets the kids on weekends then it is only 2 nights, not three nights here, 2 there, sometimes 4, etc). This will establish routine for the kids, and legally, once a pattern has been established courts don't like to change it too much. If there is not a regular patten courts may decide for you.
Has he left the house? (where you both used to live). Set firm rules about when he can and can't visit and only for the purpose of visiting the children. (no contact after 8pm, etc) and you have a friend / family member with you when this happens. (If only for a 'witness' of what he says, particularly if he is abusive, etc).
Don't talk to your ex about your depression. His lawyer may use it as ammunition. In Australia, it is very hard to get a child 'taken away' from the main care giver (basically need to be proven to be a drug dealer or in jail.) Worst case: in court he is 'seen' to be more 'stable' than you (eg no medical history on his side v any possible depression on your medical history).
In Aust there is a system so that temporary parental orders (custody) can be heard quickly in the event of family breakups and if either parent is likely to do the runner with the kids. (check if you can do similiar in NZ.) Thus if he goes with them, there's more you can do to get them back and it shows he's not willing to abide with the 'orders'. (makes it harder for him to get 'more' custody.)
I'm sorry this is long winded. I've just been through it and the above is only some of it. I always thought that I would have to do this or that, but don't hesitate to protect the kids.
You can email me if you like: [email protected]